{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d8737a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "%pip install lighteval==0.6.2\n",
    "%pip install great-tables\n",
    "%pip install polars"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d9ea816",
   "metadata": {},
   "source": [
    "# 同一任务不同 prompt 的形式对比\n",
    "这个 notebook 我们将使用一个非常小的模型来处理一个简单任务，以此为例，重点是比较输入 prompt 形式的不同对结果的影响差异。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3684eec7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "import os\n",
    "from datetime import timedelta\n",
    "from types import ModuleType\n",
    "from ast import literal_eval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a241de50",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 数据可视化\n",
    "from great_tables import GT\n",
    "import polars as pl\n",
    "import polars.selectors as cs\n",
    "from datasets import load_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e522341e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 评估结果\n",
    "import lighteval\n",
    "from lighteval.logging.evaluation_tracker import EvaluationTracker\n",
    "from lighteval.models.model_config import BaseModelConfig, VLLMModelConfig\n",
    "from lighteval.pipeline import ParallelismManager, Pipeline, PipelineParameters\n",
    "from lighteval.metrics.metrics import Metrics\n",
    "from lighteval.tasks.lighteval_task import LightevalTaskConfig, Doc\n",
    "from lighteval.utils.utils import as_list, EnvConfig\n",
    "from lighteval.utils.imports import is_accelerate_available, is_tgi_available"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d954ae1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义用例\n",
    "cache_dir = \"tmp\"\n",
    "max_samples = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a52f8c1b",
   "metadata": {},
   "source": [
    "## 同一任务多种形式对比 \n",
    "\n",
    "例如：\n",
    "- MCQA (多选问答) 评估\n",
    "- 生成式评估\n",
    "\n",
    "为比较两种评估方式，采用相同 prompt 的不同变体。\n",
    "\n",
    "本文基于 AI2 的 ARC 数据集子集进行实验。可以先点击这个链接浏览一下数据集的样子：https://huggingface.co/datasets/allenai/ai2_arc?row=0."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fc0902b",
   "metadata": {},
   "source": [
    "### 定义任务逻辑核心"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6e1c0cde",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ArcExplorationTask(LightevalTaskConfig):\n",
    "    def __init__(self, name, prompt_function, metric):\n",
    "        super().__init__(\n",
    "            name=name,\n",
    "            prompt_function=prompt_function,\n",
    "            metric=as_list(metric),\n",
    "            # 自定义任务\n",
    "            suite=[\"custom\"],\n",
    "            # 定义数据集子集\n",
    "            hf_repo=\"allenai/ai2_arc\",\n",
    "            hf_subset=\"ARC-Challenge\",\n",
    "            hf_avail_splits=[\"train\", \"validation\", \"test\"],\n",
    "            evaluation_splits=[\"test\"],\n",
    "            # few shot 选取方式参数\n",
    "            few_shots_split=\"validation\",\n",
    "            few_shots_select=\"random\", \n",
    "            # 其他任务参数\n",
    "            stop_sequence=[\".\", \"\\n\"],\n",
    "            generation_size=100,\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaa0a65a",
   "metadata": {},
   "source": [
    "### 定义度量指标\n",
    "\n",
    "对于多选评估任务，我们需要计算对数似然准确率，并按长度进行归一化 (即似然最大的选项是否为正确答案？)。\n",
    "\n",
    "对于生成式评估任务，我们需要检查精确匹配 (即生成文本与参考文本是否一致？)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "426c2ef5",
   "metadata": {},
   "outputs": [],
   "source": [
    "metric_mcqa = Metrics.loglikelihood_acc_norm\n",
    "metric_gen = Metrics.quasi_exact_match"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aea37f61",
   "metadata": {},
   "source": [
    "### 定义 prompt 函数\n",
    "\n",
    "ARC 数据集样本的格式是字典，结构如下：\n",
    "```python\n",
    "{\n",
    "    \"question\": \"the prompt question\",\n",
    "    \"choices\": {\n",
    "        \"text\": [\"choice 1\", \"choice 2\", ...],\n",
    "        \"label\": [\"A\", \"B\", ...]\n",
    "    },\n",
    "    \"answerKey\": \"the gold label\"\n",
    "}\n",
    "```\n",
    "\n",
    "我们需要定义一个 prompt 函数，将上述样本信息映射为一个模板，包括所有的请求键 (如查询、选项、标准答案索引、指令等)。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e994698",
   "metadata": {},
   "source": [
    "首先，定义最基本的模板。\n",
    "Prompt 示例：\n",
    "```\n",
    "<the question?>\n",
    "```\n",
    "直接看 `<the choices>` 选项。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "41bccce4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def arc_base(line, task_name: str = None):\n",
    "    query= f\"{line['question']}\"\n",
    "    choices=line[\"choices\"][\"text\"]\n",
    "\n",
    "    return Doc(\n",
    "        task_name=task_name,\n",
    "        query=query,\n",
    "        choices=choices,\n",
    "        gold_index=line[\"choices\"][\"label\"].index(line[\"answerKey\"]),\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d87631d5",
   "metadata": {},
   "source": [
    "其次，添加一些上下文。\n",
    "Prompt 示例：\n",
    "```\n",
    "Question: <the question?>\n",
    "Answer: \n",
    "```\n",
    "同样直接看 `<the choices>` 选项。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9865eafe",
   "metadata": {},
   "outputs": [],
   "source": [
    "def arc_context(line, task_name: str = None):\n",
    "    query= f\"Question: {line['question']}\"\n",
    "    query += \"\\nAnswer: \"\n",
    "    choices=line[\"choices\"][\"text\"]\n",
    "    return Doc(\n",
    "        task_name=task_name,\n",
    "        query=query,\n",
    "        choices=choices,\n",
    "        gold_index=line[\"choices\"][\"label\"].index(line[\"answerKey\"]),\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "132deef3",
   "metadata": {},
   "source": [
    "然后，添加选项。\n",
    "Prompt 示例：\n",
    "```\n",
    "Question: <the question?>\n",
    "Choices:\n",
    "A. <choice A>\n",
    "B. <choice B>\n",
    "...\n",
    "Answer: \n",
    "```\n",
    "继续直接看 `<the choices>` 选项。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "698367f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "letters = list(string.ascii_uppercase)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e7073026",
   "metadata": {},
   "outputs": [],
   "source": [
    "def arc_context_choices(line, task_name: str = None):\n",
    "    query = f\"Question: {line['question']}\\n\"\n",
    "    query += \"\\n\".join([f\"{letters[ix]}. {choice}\" for ix, choice in enumerate(line[\"choices\"][\"text\"])])\n",
    "    query += \"\\nAnswer: \"\n",
    "    choices=line[\"choices\"][\"text\"]\n",
    "    return Doc(\n",
    "        task_name=task_name,\n",
    "        query=query,\n",
    "        choices=choices,\n",
    "        gold_index=line[\"choices\"][\"label\"].index(line[\"answerKey\"]),\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b835822d",
   "metadata": {},
   "source": [
    "最后，我们做同样的操作，但改为查看 `<the choice labels>`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b9453b39",
   "metadata": {},
   "outputs": [],
   "source": [
    "def arc_context_labels(line, task_name: str = None):\n",
    "    query = f\"Question: {line['question']}\\n\"\n",
    "    query += \"\\n\".join([f\"{letters[ix]}. {choice}\" for ix, choice in enumerate(line[\"choices\"][\"text\"])])\n",
    "    query += \"\\nAnswer: \"\n",
    "    choices=[letters[ix] for ix in range(len(line[\"choices\"][\"text\"]))]\n",
    "    return Doc(\n",
    "        task_name=task_name,\n",
    "        query=query,\n",
    "        choices=choices,\n",
    "        gold_index=line[\"choices\"][\"label\"].index(line[\"answerKey\"]),\n",
    "    )\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da018e16",
   "metadata": {},
   "source": [
    "### `<the choice labels>`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "304a74dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_module = ModuleType(\"task_module\")\n",
    "task_module.__file__ = \".\",\n",
    "task_module.TASKS_TABLE = [\n",
    "    ArcExplorationTask(\n",
    "        name=\"arc_base\", \n",
    "        prompt_function=arc_base, \n",
    "        metric=[metric_mcqa, metric_gen]\n",
    "    ),\n",
    "    ArcExplorationTask(\n",
    "        name=\"arc_context\", \n",
    "        prompt_function=arc_context, \n",
    "        metric=[metric_mcqa, metric_gen]\n",
    "    ),\n",
    "    ArcExplorationTask(\n",
    "        name=\"arc_context_choice\", \n",
    "        prompt_function=arc_context_choices, \n",
    "        metric=[metric_mcqa, metric_gen]\n",
    "    ),\n",
    "    ArcExplorationTask(\n",
    "        name=\"arc_context_labels\", \n",
    "        prompt_function=arc_context_labels, \n",
    "        metric=[metric_mcqa, metric_gen]\n",
    "    )\n",
    "]\n",
    "\n",
    "task_names = [\"arc_base\", \"arc_context\", \"arc_context_choice\", \"arc_context_labels\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42a131fd",
   "metadata": {},
   "source": [
    "# 开始评估！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "756566ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "if is_accelerate_available():\n",
    "    from accelerate import Accelerator, InitProcessGroupKwargs\n",
    "    accelerator = Accelerator(kwargs_handlers=[InitProcessGroupKwargs(timeout=timedelta(seconds=3000))])\n",
    "else:\n",
    "    accelerator = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "1ee62d6f",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:lighteval.logging.hierarchical_logger:WARNING: --max_samples WAS SET. THESE NUMBERS ARE ONLY PARTIAL AND SHOULD NOT BE USED FOR COMPARISON UNLESS YOU KNOW WHAT YOU ARE DOING.\n",
      "WARNING:lighteval.logging.hierarchical_logger:Test all gather {\n",
      "WARNING:lighteval.logging.hierarchical_logger:  Test gather tensor\n",
      "WARNING:lighteval.logging.hierarchical_logger:  gathered_tensor tensor([0]), should be [0]\n",
      "WARNING:lighteval.logging.hierarchical_logger:} [0:00:00.002244]\n",
      "WARNING:lighteval.logging.hierarchical_logger:Model loading {\n",
      "WARNING:lighteval.logging.hierarchical_logger:  Tokenizer truncation and padding size set to the left side.\n",
      "WARNING:lighteval.logging.hierarchical_logger:  We are not in a distributed setting. Setting model_parallel to False.\n",
      "WARNING:lighteval.logging.hierarchical_logger:  Model parallel was set to False, max memory set to None and device map to None\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4e00c4d6763240ed85bde9c5f8f6614e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:lighteval.logging.hierarchical_logger:  Using Data Parallelism, putting model on device cpu\n",
      "WARNING:lighteval.logging.hierarchical_logger:} [0:00:01.973031]\n",
      "WARNING:lighteval.logging.hierarchical_logger:Tasks loading {\n",
      "WARNING:lighteval.logging.hierarchical_logger:  \u001b[33mIf you want to use extended_tasks, make sure you installed their dependencies using `pip install -e .[extended_tasks]`.\u001b[0m\n",
      "WARNING:lighteval.logging.hierarchical_logger:  {'custom|arc_base': <class 'lighteval.tasks.registry.create_config_tasks.<locals>.create_task.<locals>.LightevalTaskFromConfig'>, 'custom|arc_context': <class 'lighteval.tasks.registry.create_config_tasks.<locals>.create_task.<locals>.LightevalTaskFromConfig'>, 'custom|arc_context_choice': <class 'lighteval.tasks.registry.create_config_tasks.<locals>.create_task.<locals>.LightevalTaskFromConfig'>, 'custom|arc_context_labels': <class 'lighteval.tasks.registry.create_config_tasks.<locals>.create_task.<locals>.LightevalTaskFromConfig'>}\n",
      "WARNING:lighteval.logging.hierarchical_logger:  allenai/ai2_arc ARC-Challenge\n",
      "WARNING:lighteval.logging.hierarchical_logger:  allenai/ai2_arc ARC-Challenge\n",
      "WARNING:lighteval.logging.hierarchical_logger:  allenai/ai2_arc ARC-Challenge\n",
      "WARNING:lighteval.logging.hierarchical_logger:  allenai/ai2_arc ARC-Challenge\n",
      "WARNING:lighteval.logging.hierarchical_logger:  Loading documents, and requests\n",
      "WARNING:lighteval.logging.hierarchical_logger:} [0:00:12.300826]\n",
      "WARNING:lighteval.logging.hierarchical_logger:Setting seeds and waiting for all processes {\n",
      "WARNING:lighteval.logging.hierarchical_logger:  setting seed to 1234 for random and numpy\n",
      "WARNING:lighteval.logging.hierarchical_logger:} [0:00:00.000334]\n",
      "WARNING:lighteval.logging.hierarchical_logger:Evaluation {\n",
      "WARNING:lighteval.logging.hierarchical_logger:  Evaluate on 4 tasks.\n",
      "WARNING:lighteval.logging.hierarchical_logger:  Running RequestType.GREEDY_UNTIL requests\n",
      "WARNING:lighteval.logging.hierarchical_logger:  \u001b[33mYou cannot select the number of dataset splits for a generative evaluation at the moment. Automatically inferring.\u001b[0m\n",
      "Greedy generation: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [06:41<00:00,  9.73s/it]\u001b[A\n",
      "Splits: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [06:41<00:00, 401.95s/it]\u001b[A\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:lighteval.logging.hierarchical_logger:  Running RequestType.LOGLIKELIHOOD requests\n",
      "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [05:26<00:00,  8.17s/it]\u001b[A\n",
      "1it [05:26, 326.90s/it]\n",
      "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [04:50<00:00,  7.25s/it]\u001b[A\n",
      "2it [10:16, 305.21s/it]\n",
      "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [03:27<00:00,  5.19s/it]\u001b[A\n",
      "3it [13:44, 260.60s/it]\n",
      "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 36/36 [02:56<00:00,  4.91s/it]\u001b[A\n",
      "4it [16:41, 250.28s/it]\n",
      "WARNING:lighteval.logging.hierarchical_logger:} [0:23:23.253299]\n",
      "WARNING:lighteval.logging.hierarchical_logger:Compiling results {\n",
      "WARNING:lighteval.logging.hierarchical_logger:} [0:00:00.000472]\n",
      "WARNING:lighteval.logging.hierarchical_logger:Cleaning up {\n",
      "WARNING:lighteval.logging.hierarchical_logger:} [0:00:00.000034]\n",
      "WARNING:lighteval.logging.hierarchical_logger:Saving experiment tracker\n",
      "WARNING:lighteval.logging.hierarchical_logger:Saving results to .../tmp/results/HuggingFaceTB/SmolLM-1.7B/results_2024-10-24T16-24-58.398434.json\n",
      "WARNING:lighteval.logging.hierarchical_logger:Saving details to .../tmp/details/HuggingFaceTB/SmolLM-1.7B/2024-10-24T16-24-58.398434\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b1e4e357b68b41f8ac49469cddda318d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating parquet from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "434138b2f5f84ffcae32685dcf71a6c7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating parquet from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9200001cbdc6422197dc65aaacebe6b4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating parquet from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "685264a9da3b4c79b1fef669cec70460",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating parquet from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 参数设置：保存评估结果\n",
    "evaluation_tracker = EvaluationTracker(\n",
    "    output_dir=cache_dir,\n",
    "    save_details=True,\n",
    "    # 下列两个参数需要使用 Huggingface_hub CLI 登录后才能启用\n",
    "    #push_to_hub=True,\n",
    "    #hub_results_org=\"your username\", \n",
    ")\n",
    "\n",
    "\n",
    "# 参数设置：整体 Pipeline\n",
    "pipeline_params = PipelineParameters(\n",
    "    launcher_type=ParallelismManager.ACCELERATE,\n",
    "    env_config=EnvConfig(cache_dir=cache_dir),\n",
    "    override_batch_size=1,\n",
    "    max_samples=max_samples, # 本文仅小批量测试，如需获取全部结果请注释该参数 \n",
    "    custom_tasks_directory=task_module # 该参数可以传模块路径或模块实例\n",
    ")\n",
    "\n",
    "# 模型设置：本文使用 VLLM，你也可以使用 TGI、 Accelerate 等\n",
    "model_config = BaseModelConfig(\n",
    "    pretrained=\"HuggingFaceTB/SmolLM-1.7B\",\n",
    "    dtype=\"bfloat16\",\n",
    "    use_chat_template=False,\n",
    ")\n",
    "\n",
    "tasks = \",\".join([f\"custom|{task}|3|0\" for task in task_names])\n",
    "# 开始运行!\n",
    "pipeline = Pipeline(\n",
    "    tasks=tasks,\n",
    "    pipeline_parameters=pipeline_params,\n",
    "    evaluation_tracker=evaluation_tracker,\n",
    "    model_config=model_config,\n",
    ")\n",
    "pipeline.evaluate()\n",
    "pipeline.save_and_push_results()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "38b63709",
   "metadata": {},
   "outputs": [],
   "source": [
    "#pipeline.show_results()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8ccb72d2",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"zlaxyprctg\" style=\"padding-left:0px;padding-right:0px;padding-top:10px;padding-bottom:10px;overflow-x:auto;overflow-y:auto;width:auto;height:auto;\">\n",
       "<style>\n",
       "#zlaxyprctg table {\n",
       "          font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;\n",
       "          -webkit-font-smoothing: antialiased;\n",
       "          -moz-osx-font-smoothing: grayscale;\n",
       "        }\n",
       "\n",
       "#zlaxyprctg thead, tbody, tfoot, tr, td, th { border-style: none; }\n",
       " tr { background-color: transparent; }\n",
       "#zlaxyprctg p { margin: 0; padding: 0; }\n",
       " #zlaxyprctg .gt_table { display: table; border-collapse: collapse; line-height: normal; margin-left: auto; margin-right: auto; color: #333333; font-size: 16px; font-weight: normal; font-style: normal; background-color: #FFFFFF; width: auto; border-top-style: solid; border-top-width: 2px; border-top-color: #A8A8A8; border-right-style: none; border-right-width: 2px; border-right-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #A8A8A8; border-left-style: none; border-left-width: 2px; border-left-color: #D3D3D3; }\n",
       " #zlaxyprctg .gt_caption { padding-top: 4px; padding-bottom: 4px; }\n",
       " #zlaxyprctg .gt_title { color: #333333; font-size: 125%; font-weight: initial; padding-top: 4px; padding-bottom: 4px; padding-left: 5px; padding-right: 5px; border-bottom-color: #FFFFFF; border-bottom-width: 0; }\n",
       " #zlaxyprctg .gt_subtitle { color: #333333; font-size: 85%; font-weight: initial; padding-top: 3px; padding-bottom: 5px; padding-left: 5px; padding-right: 5px; border-top-color: #FFFFFF; border-top-width: 0; }\n",
       " #zlaxyprctg .gt_heading { background-color: #FFFFFF; text-align: center; border-bottom-color: #FFFFFF; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; }\n",
       " #zlaxyprctg .gt_bottom_border { border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; }\n",
       " #zlaxyprctg .gt_col_headings { border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; }\n",
       " #zlaxyprctg .gt_col_heading { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: normal; text-transform: inherit; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; vertical-align: bottom; padding-top: 5px; padding-bottom: 5px; padding-left: 5px; padding-right: 5px; overflow-x: hidden; }\n",
       " #zlaxyprctg .gt_column_spanner_outer { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: normal; text-transform: inherit; padding-top: 0; padding-bottom: 0; padding-left: 4px; padding-right: 4px; }\n",
       " #zlaxyprctg .gt_column_spanner_outer:first-child { padding-left: 0; }\n",
       " #zlaxyprctg .gt_column_spanner_outer:last-child { padding-right: 0; }\n",
       " #zlaxyprctg .gt_column_spanner { border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; vertical-align: bottom; padding-top: 5px; padding-bottom: 5px; overflow-x: hidden; display: inline-block; width: 100%; }\n",
       " #zlaxyprctg .gt_spanner_row { border-bottom-style: hidden; }\n",
       " #zlaxyprctg .gt_group_heading { padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px; color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; text-transform: inherit; border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; vertical-align: middle; text-align: left; }\n",
       " #zlaxyprctg .gt_empty_group_heading { padding: 0.5px; color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; vertical-align: middle; }\n",
       " #zlaxyprctg .gt_from_md> :first-child { margin-top: 0; }\n",
       " #zlaxyprctg .gt_from_md> :last-child { margin-bottom: 0; }\n",
       " #zlaxyprctg .gt_row { padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px; margin: 10px; border-top-style: solid; border-top-width: 1px; border-top-color: #D3D3D3; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; vertical-align: middle; overflow-x: hidden; }\n",
       " #zlaxyprctg .gt_stub { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; text-transform: inherit; border-right-style: solid; border-right-width: 2px; border-right-color: #D3D3D3; padding-left: 5px; padding-right: 5px; }\n",
       " #zlaxyprctg .gt_stub_row_group { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; text-transform: inherit; border-right-style: solid; border-right-width: 2px; border-right-color: #D3D3D3; padding-left: 5px; padding-right: 5px; vertical-align: top; }\n",
       " #zlaxyprctg .gt_row_group_first td { border-top-width: 2px; }\n",
       " #zlaxyprctg .gt_row_group_first th { border-top-width: 2px; }\n",
       " #zlaxyprctg .gt_striped { background-color: rgba(128,128,128,0.05); }\n",
       " #zlaxyprctg .gt_table_body { border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; }\n",
       " #zlaxyprctg .gt_sourcenotes { color: #333333; background-color: #FFFFFF; border-bottom-style: none; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 2px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 2px; border-right-color: #D3D3D3; }\n",
       " #zlaxyprctg .gt_sourcenote { font-size: 90%; padding-top: 4px; padding-bottom: 4px; padding-left: 5px; padding-right: 5px; text-align: left; }\n",
       " #zlaxyprctg .gt_left { text-align: left; }\n",
       " #zlaxyprctg .gt_center { text-align: center; }\n",
       " #zlaxyprctg .gt_right { text-align: right; font-variant-numeric: tabular-nums; }\n",
       " #zlaxyprctg .gt_font_normal { font-weight: normal; }\n",
       " #zlaxyprctg .gt_font_bold { font-weight: bold; }\n",
       " #zlaxyprctg .gt_font_italic { font-style: italic; }\n",
       " #zlaxyprctg .gt_super { font-size: 65%; }\n",
       " #zlaxyprctg .gt_footnote_marks { font-size: 75%; vertical-align: 0.4em; position: initial; }\n",
       " #zlaxyprctg .gt_asterisk { font-size: 100%; vertical-align: 0; }\n",
       " \n",
       "</style>\n",
       "<table class=\"gt_table\" data-quarto-disable-processing=\"false\" data-quarto-bootstrap=\"false\">\n",
       "<thead>\n",
       "\n",
       "  <tr class=\"gt_heading\">\n",
       "    <td colspan=\"3\" class=\"gt_heading gt_title gt_font_normal\">Results</td>\n",
       "  </tr>\n",
       "<tr class=\"gt_col_headings gt_spanner_row\">\n",
       "  <th class=\"gt_col_heading gt_columns_bottom_border gt_left\" rowspan=\"2\" colspan=\"1\" scope=\"col\" id=\"Prompt function\">Prompt function</th>\n",
       "  <th class=\"gt_center gt_columns_top_border gt_column_spanner_outer\" rowspan=\"1\" colspan=\"2\" scope=\"colgroup\" id=\"Evaluations\">\n",
       "    <span class=\"gt_column_spanner\">Evaluations</span>\n",
       "  </th>\n",
       "</tr>\n",
       "<tr class=\"gt_col_headings\">\n",
       "  <th class=\"gt_col_heading gt_columns_bottom_border gt_right\" rowspan=\"1\" colspan=\"1\" scope=\"col\" id=\"Quasi Exact Match\">Quasi Exact Match</th>\n",
       "  <th class=\"gt_col_heading gt_columns_bottom_border gt_right\" rowspan=\"1\" colspan=\"1\" scope=\"col\" id=\"Normalized Accuracy\">Normalized Accuracy</th>\n",
       "</tr>\n",
       "</thead>\n",
       "<tbody class=\"gt_table_body\">\n",
       "  <tr>\n",
       "    <td class=\"gt_row gt_left\">arc base</td>\n",
       "    <td class=\"gt_row gt_right\">0.0</td>\n",
       "    <td class=\"gt_row gt_right\">0.3</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <td class=\"gt_row gt_left\">arc context</td>\n",
       "    <td class=\"gt_row gt_right\">0.1</td>\n",
       "    <td class=\"gt_row gt_right\">0.5</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <td class=\"gt_row gt_left\">arc context choice</td>\n",
       "    <td class=\"gt_row gt_right\">0.4</td>\n",
       "    <td class=\"gt_row gt_right\">0.3</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <td class=\"gt_row gt_left\">arc context labels</td>\n",
       "    <td class=\"gt_row gt_right\">0.0</td>\n",
       "    <td class=\"gt_row gt_right\">0.0</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <td class=\"gt_row gt_left\">all</td>\n",
       "    <td class=\"gt_row gt_right\">0.125</td>\n",
       "    <td class=\"gt_row gt_right\">0.275</td>\n",
       "  </tr>\n",
       "</tbody>\n",
       "\n",
       "\n",
       "</table>\n",
       "\n",
       "</div>\n",
       "        "
      ],
      "text/plain": [
       "GT(_tbl_data=shape: (5, 3)\n",
       "┌────────────────────┬───────────────────┬─────────────────────┐\n",
       "│ Prompt function    ┆ Quasi Exact Match ┆ Normalized Accuracy │\n",
       "│ ---                ┆ ---               ┆ ---                 │\n",
       "│ str                ┆ f64               ┆ f64                 │\n",
       "╞════════════════════╪═══════════════════╪═════════════════════╡\n",
       "│ arc base           ┆ 0.0               ┆ 0.3                 │\n",
       "│ arc context        ┆ 0.1               ┆ 0.5                 │\n",
       "│ arc context choice ┆ 0.4               ┆ 0.3                 │\n",
       "│ arc context labels ┆ 0.0               ┆ 0.0                 │\n",
       "│ all                ┆ 0.125             ┆ 0.275               │\n",
       "└────────────────────┴───────────────────┴─────────────────────┘, _body=<great_tables._gt_data.Body object at 0x7f074da27010>, _boxhead=Boxhead([ColInfo(var='Prompt function', type=<ColInfoTypeEnum.default: 1>, column_label='Prompt function', column_align='left', column_width=None), ColInfo(var='Quasi Exact Match', type=<ColInfoTypeEnum.default: 1>, column_label='Quasi Exact Match', column_align='right', column_width=None), ColInfo(var='Normalized Accuracy', type=<ColInfoTypeEnum.default: 1>, column_label='Normalized Accuracy', column_align='right', column_width=None)]), _stub=<great_tables._gt_data.Stub object at 0x7f074da277c0>, _spanners=Spanners([SpannerInfo(spanner_id='Evaluations', spanner_level=0, spanner_label='Evaluations', spanner_units=None, spanner_pattern=None, vars=['Quasi Exact Match', 'Normalized Accuracy'], built=None)]), _heading=Heading(title='Results', subtitle=None, preheader=None), _stubhead=None, _source_notes=[], _footnotes=[], _styles=[], _locale=<great_tables._gt_data.Locale object at 0x7f074c8b6e30>, _formats=[], _substitutions=[], _options=Options(table_id=OptionsInfo(scss=False, category='table', type='value', value=None), table_caption=OptionsInfo(scss=False, category='table', type='value', value=None), table_width=OptionsInfo(scss=True, category='table', type='px', value='auto'), table_layout=OptionsInfo(scss=True, category='table', type='value', value='fixed'), table_margin_left=OptionsInfo(scss=True, category='table', type='px', value='auto'), table_margin_right=OptionsInfo(scss=True, category='table', type='px', value='auto'), table_background_color=OptionsInfo(scss=True, category='table', type='value', value='#FFFFFF'), table_additional_css=OptionsInfo(scss=False, category='table', type='values', value=[]), table_font_names=OptionsInfo(scss=False, category='table', type='values', value=['-apple-system', 'BlinkMacSystemFont', 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarell', 'Helvetica Neue', 'Fira Sans', 'Droid Sans', 'Arial', 'sans-serif']), table_font_size=OptionsInfo(scss=True, category='table', type='px', value='16px'), table_font_weight=OptionsInfo(scss=True, category='table', type='value', value='normal'), table_font_style=OptionsInfo(scss=True, category='table', type='value', value='normal'), table_font_color=OptionsInfo(scss=True, category='table', type='value', value='#333333'), table_font_color_light=OptionsInfo(scss=True, category='table', type='value', value='#FFFFFF'), table_border_top_include=OptionsInfo(scss=False, category='table', type='boolean', value=True), table_border_top_style=OptionsInfo(scss=True, category='table', type='value', value='solid'), table_border_top_width=OptionsInfo(scss=True, category='table', type='px', value='2px'), table_border_top_color=OptionsInfo(scss=True, category='table', type='value', value='#A8A8A8'), table_border_right_style=OptionsInfo(scss=True, category='table', type='value', value='none'), table_border_right_width=OptionsInfo(scss=True, category='table', type='px', value='2px'), table_border_right_color=OptionsInfo(scss=True, category='table', type='value', value='#D3D3D3'), table_border_bottom_include=OptionsInfo(scss=False, category='table', type='boolean', value=True), table_border_bottom_style=OptionsInfo(scss=True, category='table', type='value', value='solid'), table_border_bottom_width=OptionsInfo(scss=True, category='table', type='px', value='2px'), table_border_bottom_color=OptionsInfo(scss=True, category='table', type='value', value='#A8A8A8'), table_border_left_style=OptionsInfo(scss=True, category='table', type='value', value='none'), table_border_left_width=OptionsInfo(scss=True, category='table', type='px', value='2px'), table_border_left_color=OptionsInfo(scss=True, category='table', type='value', value='#D3D3D3'), heading_background_color=OptionsInfo(scss=True, category='heading', type='value', value=None), heading_align=OptionsInfo(scss=True, category='heading', type='value', value='center'), heading_title_font_size=OptionsInfo(scss=True, category='heading', type='px', value='125%'), heading_title_font_weight=OptionsInfo(scss=True, category='heading', type='value', value='initial'), heading_subtitle_font_size=OptionsInfo(scss=True, category='heading', type='px', value='85%'), heading_subtitle_font_weight=OptionsInfo(scss=True, category='heading', type='value', value='initial'), heading_padding=OptionsInfo(scss=True, category='heading', type='px', value='4px'), heading_padding_horizontal=OptionsInfo(scss=True, category='heading', type='px', value='5px'), heading_border_bottom_style=OptionsInfo(scss=True, category='heading', type='value', value='solid'), heading_border_bottom_width=OptionsInfo(scss=True, category='heading', type='px', value='2px'), heading_border_bottom_color=OptionsInfo(scss=True, category='heading', type='value', value='#D3D3D3'), heading_border_lr_style=OptionsInfo(scss=True, category='heading', type='value', value='none'), heading_border_lr_width=OptionsInfo(scss=True, category='heading', type='px', value='1px'), heading_border_lr_color=OptionsInfo(scss=True, category='heading', type='value', value='#D3D3D3'), column_labels_background_color=OptionsInfo(scss=True, category='column_labels', type='value', value=None), column_labels_font_size=OptionsInfo(scss=True, category='column_labels', type='px', value='100%'), column_labels_font_weight=OptionsInfo(scss=True, category='column_labels', type='value', value='normal'), column_labels_text_transform=OptionsInfo(scss=True, category='column_labels', type='value', value='inherit'), column_labels_padding=OptionsInfo(scss=True, category='column_labels', type='px', value='5px'), column_labels_padding_horizontal=OptionsInfo(scss=True, category='column_labels', type='px', value='5px'), column_labels_vlines_style=OptionsInfo(scss=True, category='table_body', type='value', value='none'), column_labels_vlines_width=OptionsInfo(scss=True, category='table_body', type='px', value='1px'), column_labels_vlines_color=OptionsInfo(scss=True, category='table_body', type='value', value='#D3D3D3'), column_labels_border_top_style=OptionsInfo(scss=True, category='column_labels', type='value', value='solid'), column_labels_border_top_width=OptionsInfo(scss=True, category='column_labels', type='px', value='2px'), column_labels_border_top_color=OptionsInfo(scss=True, category='column_labels', type='value', value='#D3D3D3'), column_labels_border_bottom_style=OptionsInfo(scss=True, category='column_labels', type='value', value='solid'), column_labels_border_bottom_width=OptionsInfo(scss=True, category='column_labels', type='px', value='2px'), column_labels_border_bottom_color=OptionsInfo(scss=True, category='column_labels', type='value', value='#D3D3D3'), column_labels_border_lr_style=OptionsInfo(scss=True, category='column_labels', type='value', value='none'), column_labels_border_lr_width=OptionsInfo(scss=True, category='column_labels', type='px', value='1px'), column_labels_border_lr_color=OptionsInfo(scss=True, category='column_labels', type='value', value='#D3D3D3'), column_labels_hidden=OptionsInfo(scss=False, category='column_labels', type='boolean', value=False), row_group_background_color=OptionsInfo(scss=True, category='row_group', type='value', value=None), row_group_font_size=OptionsInfo(scss=True, category='row_group', type='px', value='100%'), row_group_font_weight=OptionsInfo(scss=True, category='row_group', type='value', value='initial'), row_group_text_transform=OptionsInfo(scss=True, category='row_group', type='value', value='inherit'), row_group_padding=OptionsInfo(scss=True, category='row_group', type='px', value='8px'), row_group_padding_horizontal=OptionsInfo(scss=True, category='row_group', type='px', value='5px'), row_group_border_top_style=OptionsInfo(scss=True, category='row_group', type='value', value='solid'), row_group_border_top_width=OptionsInfo(scss=True, category='row_group', type='px', value='2px'), row_group_border_top_color=OptionsInfo(scss=True, category='row_group', type='value', value='#D3D3D3'), row_group_border_right_style=OptionsInfo(scss=True, category='row_group', type='value', value='none'), row_group_border_right_width=OptionsInfo(scss=True, category='row_group', type='px', value='1px'), row_group_border_right_color=OptionsInfo(scss=True, category='row_group', type='value', value='#D3D3D3'), row_group_border_bottom_style=OptionsInfo(scss=True, category='row_group', type='value', value='solid'), row_group_border_bottom_width=OptionsInfo(scss=True, category='row_group', type='px', value='2px'), row_group_border_bottom_color=OptionsInfo(scss=True, category='row_group', type='value', value='#D3D3D3'), row_group_border_left_style=OptionsInfo(scss=True, category='row_group', type='value', value='none'), row_group_border_left_width=OptionsInfo(scss=True, category='row_group', type='px', value='1px'), row_group_border_left_color=OptionsInfo(scss=True, category='row_group', type='value', value='#D3D3D3'), row_group_as_column=OptionsInfo(scss=False, category='row_group', type='boolean', value=False), table_body_hlines_style=OptionsInfo(scss=True, category='table_body', type='value', value='solid'), table_body_hlines_width=OptionsInfo(scss=True, category='table_body', type='px', value='1px'), table_body_hlines_color=OptionsInfo(scss=True, category='table_body', type='value', value='#D3D3D3'), table_body_vlines_style=OptionsInfo(scss=True, category='table_body', type='value', value='none'), table_body_vlines_width=OptionsInfo(scss=True, category='table_body', type='px', value='1px'), table_body_vlines_color=OptionsInfo(scss=True, category='table_body', type='value', value='#D3D3D3'), table_body_border_top_style=OptionsInfo(scss=True, category='table_body', type='value', value='solid'), table_body_border_top_width=OptionsInfo(scss=True, category='table_body', type='px', value='2px'), table_body_border_top_color=OptionsInfo(scss=True, category='table_body', type='value', value='#D3D3D3'), table_body_border_bottom_style=OptionsInfo(scss=True, category='table_body', type='value', value='solid'), table_body_border_bottom_width=OptionsInfo(scss=True, category='table_body', type='px', value='2px'), table_body_border_bottom_color=OptionsInfo(scss=True, category='table_body', type='value', value='#D3D3D3'), data_row_padding=OptionsInfo(scss=True, category='data_row', type='px', value='8px'), data_row_padding_horizontal=OptionsInfo(scss=True, category='data_row', type='px', value='5px'), stub_background_color=OptionsInfo(scss=True, category='stub', type='value', value=None), stub_font_size=OptionsInfo(scss=True, category='stub', type='px', value='100%'), stub_font_weight=OptionsInfo(scss=True, category='stub', type='value', value='initial'), stub_text_transform=OptionsInfo(scss=True, category='stub', type='value', value='inherit'), stub_border_style=OptionsInfo(scss=True, category='stub', type='value', value='solid'), stub_border_width=OptionsInfo(scss=True, category='stub', type='px', value='2px'), stub_border_color=OptionsInfo(scss=True, category='stub', type='value', value='#D3D3D3'), stub_row_group_background_color=OptionsInfo(scss=True, category='stub', type='value', value=None), stub_row_group_font_size=OptionsInfo(scss=True, category='stub', type='px', value='100%'), stub_row_group_font_weight=OptionsInfo(scss=True, category='stub', type='value', value='initial'), stub_row_group_text_transform=OptionsInfo(scss=True, category='stub', type='value', value='inherit'), stub_row_group_border_style=OptionsInfo(scss=True, category='stub', type='value', value='solid'), stub_row_group_border_width=OptionsInfo(scss=True, category='stub', type='px', value='2px'), stub_row_group_border_color=OptionsInfo(scss=True, category='stub', type='value', value='#D3D3D3'), source_notes_padding=OptionsInfo(scss=True, category='source_notes', type='px', value='4px'), source_notes_padding_horizontal=OptionsInfo(scss=True, category='source_notes', type='px', value='5px'), source_notes_background_color=OptionsInfo(scss=True, category='source_notes', type='value', value=None), source_notes_font_size=OptionsInfo(scss=True, category='source_notes', type='px', value='90%'), source_notes_border_bottom_style=OptionsInfo(scss=True, category='source_notes', type='value', value='none'), source_notes_border_bottom_width=OptionsInfo(scss=True, category='source_notes', type='px', value='2px'), source_notes_border_bottom_color=OptionsInfo(scss=True, category='source_notes', type='value', value='#D3D3D3'), source_notes_border_lr_style=OptionsInfo(scss=True, category='source_notes', type='value', value='none'), source_notes_border_lr_width=OptionsInfo(scss=True, category='source_notes', type='px', value='2px'), source_notes_border_lr_color=OptionsInfo(scss=True, category='source_notes', type='value', value='#D3D3D3'), source_notes_multiline=OptionsInfo(scss=False, category='source_notes', type='boolean', value=True), source_notes_sep=OptionsInfo(scss=False, category='source_notes', type='value', value=' '), row_striping_background_color=OptionsInfo(scss=True, category='row', type='value', value='rgba(128,128,128,0.05)'), row_striping_include_stub=OptionsInfo(scss=False, category='row', type='boolean', value=False), row_striping_include_table_body=OptionsInfo(scss=False, category='row', type='boolean', value=False), container_width=OptionsInfo(scss=False, category='container', type='px', value='auto'), container_height=OptionsInfo(scss=False, category='container', type='px', value='auto'), container_padding_x=OptionsInfo(scss=False, category='container', type='px', value='0px'), container_padding_y=OptionsInfo(scss=False, category='container', type='px', value='10px'), container_overflow_x=OptionsInfo(scss=False, category='container', type='overflow', value='auto'), container_overflow_y=OptionsInfo(scss=False, category='container', type='overflow', value='auto'), quarto_disable_processing=OptionsInfo(scss=False, category='quarto', type='logical', value=False), quarto_use_bootstrap=OptionsInfo(scss=False, category='quarto', type='logical', value=False)), _has_built=False)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = pipeline.get_results()[\"results\"]\n",
    "results_processed = []\n",
    "for eval_name, eval_results in results.items():\n",
    "    results_processed.append({\n",
    "        \"Prompt function\": (eval_name.split(\":\")[1] if \":\" in eval_name else eval_name).replace(\"_\", \" \"), \n",
    "        \"Quasi Exact Match\": eval_results[\"qem\"], \n",
    "        \"Normalized Accuracy\": eval_results[\"acc_norm\"]\n",
    "    })\n",
    "results_data = pl.from_dicts(results_processed, strict=False)\n",
    "(GT(results_data.head(max_samples*4))\n",
    "    .tab_header(\"Results\")\n",
    "     .tab_spanner(label=\"Evaluations\", columns=[\"Quasi Exact Match\", \"Normalized Accuracy\"])\n",
    "\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "872c4403",
   "metadata": {},
   "source": [
    "# 读取结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f4aae325",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2ca65572fde9462ab6c29b6f17974c65",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "84e8f2c884de4ecb9b9e2d4c158128a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6dfe668ee83242fb9b9070b875db3f40",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "11ff92775dca4ff5aca86c24fccc038b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "path = f\"{cache_dir}/details/HuggingFaceTB/SmolLM-1.7B/\"\n",
    "\n",
    "results = {}\n",
    "\n",
    "for root, _, files in os.walk(path):\n",
    "    for file in files:\n",
    "        eval_name = file.split(\"|\")[1]\n",
    "        results[eval_name] = load_dataset(\"parquet\", data_files=f\"{root}/{file}\")[\"train\"]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "08165452",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 新建一个 DataFrame 来存储转换数据\n",
    "transformed_data = []\n",
    "keys = [\"example\", \"gold\", \"predictions\", \"metrics\"]\n",
    "\n",
    "# 遍历所有数据集和样本\n",
    "for ix in range(max_samples * 2):\n",
    "    for key in keys:\n",
    "        cur_sample = {\"Sample\": f\"Sample {ix}\", \"Type\": key.capitalize()}\n",
    "        for eval_name, df in sorted(results.items()):\n",
    "            try:\n",
    "                cur_result = literal_eval(results[eval_name][ix][key])\n",
    "                if isinstance(cur_result, list):\n",
    "                    if len(cur_result) == 1:\n",
    "                        cur_sample[eval_name] = cur_result[0]\n",
    "                    else:\n",
    "                        cur_sample[eval_name] = \"\\n\".join([str(i) for i in cur_result])\n",
    "                elif isinstance(cur_result, dict):\n",
    "                    for metric, value in cur_result.items():\n",
    "                        cur_sample[eval_name] = str(value)\n",
    "                        cur_sample[\"Type\"] = f\"{key.capitalize()}: {metric}\"\n",
    "            except SyntaxError:\n",
    "                cur_sample[eval_name] = results[eval_name][ix][key]\n",
    "                \n",
    "        for k, v in cur_sample.items():\n",
    "            # 将 Python 中的换行符转换为 Markdown 格式\n",
    "            if isinstance(v, str):\n",
    "                cur_sample[k] = v.replace(\"\\n\", \"<br />\")\n",
    "        transformed_data.append(cur_sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2f3d2f2",
   "metadata": {},
   "source": [
    "### 可视化生成结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "25cde03f",
   "metadata": {},
   "outputs": [],
   "source": [
    "pl_data = pl.from_dicts(transformed_data, strict=False, infer_schema_length=200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "bcd7288e",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"auiqqdecam\" style=\"padding-left:0px;padding-right:0px;padding-top:10px;padding-bottom:10px;overflow-x:auto;overflow-y:auto;width:auto;height:auto;\">\n",
       "<style>\n",
       "#auiqqdecam table {\n",
       "          font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;\n",
       "          -webkit-font-smoothing: antialiased;\n",
       "          -moz-osx-font-smoothing: grayscale;\n",
       "        }\n",
       "\n",
       "#auiqqdecam thead, tbody, tfoot, tr, td, th { border-style: none; }\n",
       " tr { background-color: transparent; }\n",
       "#auiqqdecam p { margin: 0; padding: 0; }\n",
       " #auiqqdecam .gt_table { display: table; border-collapse: collapse; line-height: normal; margin-left: auto; margin-right: auto; color: #333333; font-size: 16px; font-weight: normal; font-style: normal; background-color: #FFFFFF; width: auto; border-top-style: solid; border-top-width: 2px; border-top-color: #A8A8A8; border-right-style: none; border-right-width: 2px; border-right-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #A8A8A8; border-left-style: none; border-left-width: 2px; border-left-color: #D3D3D3; }\n",
       " #auiqqdecam .gt_caption { padding-top: 4px; padding-bottom: 4px; }\n",
       " #auiqqdecam .gt_title { color: #333333; font-size: 125%; font-weight: initial; padding-top: 4px; padding-bottom: 4px; padding-left: 5px; padding-right: 5px; border-bottom-color: #FFFFFF; border-bottom-width: 0; }\n",
       " #auiqqdecam .gt_subtitle { color: #333333; font-size: 85%; font-weight: initial; padding-top: 3px; padding-bottom: 5px; padding-left: 5px; padding-right: 5px; border-top-color: #FFFFFF; border-top-width: 0; }\n",
       " #auiqqdecam .gt_heading { background-color: #FFFFFF; text-align: center; border-bottom-color: #FFFFFF; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; }\n",
       " #auiqqdecam .gt_bottom_border { border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; }\n",
       " #auiqqdecam .gt_col_headings { border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; }\n",
       " #auiqqdecam .gt_col_heading { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: normal; text-transform: inherit; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; vertical-align: bottom; padding-top: 5px; padding-bottom: 5px; padding-left: 5px; padding-right: 5px; overflow-x: hidden; }\n",
       " #auiqqdecam .gt_column_spanner_outer { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: normal; text-transform: inherit; padding-top: 0; padding-bottom: 0; padding-left: 4px; padding-right: 4px; }\n",
       " #auiqqdecam .gt_column_spanner_outer:first-child { padding-left: 0; }\n",
       " #auiqqdecam .gt_column_spanner_outer:last-child { padding-right: 0; }\n",
       " #auiqqdecam .gt_column_spanner { border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; vertical-align: bottom; padding-top: 5px; padding-bottom: 5px; overflow-x: hidden; display: inline-block; width: 100%; }\n",
       " #auiqqdecam .gt_spanner_row { border-bottom-style: hidden; }\n",
       " #auiqqdecam .gt_group_heading { padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px; color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; text-transform: inherit; border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; vertical-align: middle; text-align: left; }\n",
       " #auiqqdecam .gt_empty_group_heading { padding: 0.5px; color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; vertical-align: middle; }\n",
       " #auiqqdecam .gt_from_md> :first-child { margin-top: 0; }\n",
       " #auiqqdecam .gt_from_md> :last-child { margin-bottom: 0; }\n",
       " #auiqqdecam .gt_row { padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px; margin: 10px; border-top-style: solid; border-top-width: 1px; border-top-color: #D3D3D3; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; vertical-align: middle; overflow-x: hidden; }\n",
       " #auiqqdecam .gt_stub { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; text-transform: inherit; border-right-style: solid; border-right-width: 2px; border-right-color: #D3D3D3; padding-left: 5px; padding-right: 5px; }\n",
       " #auiqqdecam .gt_stub_row_group { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; text-transform: inherit; border-right-style: solid; border-right-width: 2px; border-right-color: #D3D3D3; padding-left: 5px; padding-right: 5px; vertical-align: top; }\n",
       " #auiqqdecam .gt_row_group_first td { border-top-width: 2px; }\n",
       " #auiqqdecam .gt_row_group_first th { border-top-width: 2px; }\n",
       " #auiqqdecam .gt_striped { background-color: rgba(128,128,128,0.05); }\n",
       " #auiqqdecam .gt_table_body { border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; }\n",
       " #auiqqdecam .gt_sourcenotes { color: #333333; background-color: #FFFFFF; border-bottom-style: none; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 2px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 2px; border-right-color: #D3D3D3; }\n",
       " #auiqqdecam .gt_sourcenote { font-size: 90%; padding-top: 4px; padding-bottom: 4px; padding-left: 5px; padding-right: 5px; text-align: left; }\n",
       " #auiqqdecam .gt_left { text-align: left; }\n",
       " #auiqqdecam .gt_center { text-align: center; }\n",
       " #auiqqdecam .gt_right { text-align: right; font-variant-numeric: tabular-nums; }\n",
       " #auiqqdecam .gt_font_normal { font-weight: normal; }\n",
       " #auiqqdecam .gt_font_bold { font-weight: bold; }\n",
       " #auiqqdecam .gt_font_italic { font-style: italic; }\n",
       " #auiqqdecam .gt_super { font-size: 65%; }\n",
       " #auiqqdecam .gt_footnote_marks { font-size: 75%; vertical-align: 0.4em; position: initial; }\n",
       " #auiqqdecam .gt_asterisk { font-size: 100%; vertical-align: 0; }\n",
       " \n",
       "</style>\n",
       "<table class=\"gt_table\" data-quarto-disable-processing=\"false\" data-quarto-bootstrap=\"false\">\n",
       "<thead>\n",
       "\n",
       "  <tr class=\"gt_heading\">\n",
       "    <td colspan=\"5\" class=\"gt_heading gt_title gt_font_normal\">Comparing our different prompts&#x27; outputs</td>\n",
       "  </tr>\n",
       "<tr class=\"gt_col_headings gt_spanner_row\">\n",
       "  <th class=\"gt_col_heading gt_columns_bottom_border gt_left\" rowspan=\"2\" colspan=\"1\" scope=\"col\" id=\"\"></th>\n",
       "  <th class=\"gt_center gt_columns_top_border gt_column_spanner_outer\" rowspan=\"1\" colspan=\"4\" scope=\"colgroup\" id=\"Samples\">\n",
       "    <span class=\"gt_column_spanner\">Samples</span>\n",
       "  </th>\n",
       "</tr>\n",
       "<tr class=\"gt_col_headings\">\n",
       "  <th class=\"gt_col_heading gt_columns_bottom_border gt_left\" rowspan=\"1\" colspan=\"1\" scope=\"col\" id=\"arc_base\">arc_base</th>\n",
       "  <th class=\"gt_col_heading gt_columns_bottom_border gt_left\" rowspan=\"1\" colspan=\"1\" scope=\"col\" id=\"arc_context\">arc_context</th>\n",
       "  <th class=\"gt_col_heading gt_columns_bottom_border gt_left\" rowspan=\"1\" colspan=\"1\" scope=\"col\" id=\"arc_context_choice\">arc_context_choice</th>\n",
       "  <th class=\"gt_col_heading gt_columns_bottom_border gt_left\" rowspan=\"1\" colspan=\"1\" scope=\"col\" id=\"arc_context_labels\">arc_context_labels</th>\n",
       "</tr>\n",
       "</thead>\n",
       "<tbody class=\"gt_table_body\">\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 0</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">Cities control the amount of pollution that is allowed to come from cars. How does this most likely help people?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Cities control the amount of pollution that is allowed to come from cars. How does this most likely help people?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Cities control the amount of pollution that is allowed to come from cars. How does this most likely help people?<br />A. The air stays cleaner.<br />B. Cars can travel at faster speeds.<br />C. The skills of the drivers improve.<br />D. It becomes safer to drive on the roads.<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Cities control the amount of pollution that is allowed to come from cars. How does this most likely help people?<br />A. The air stays cleaner.<br />B. Cars can travel at faster speeds.<br />C. The skills of the drivers improve.<br />D. It becomes safer to drive on the roads.<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">The air stays cleaner.</td>\n",
       "    <td class=\"gt_row gt_left\">The air stays cleaner.</td>\n",
       "    <td class=\"gt_row gt_left\">The air stays cleaner.</td>\n",
       "    <td class=\"gt_row gt_left\">A</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">It reduces the amount of pollution in the air</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">It becomes safer to drive on the roads</td>\n",
       "    <td class=\"gt_row gt_left\">D</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: qem</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 1</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">Which statement correctly describes a physical characteristic of the Moon?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Which statement correctly describes a physical characteristic of the Moon?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Which statement correctly describes a physical characteristic of the Moon?<br />A. The Moon is made of hot gases.<br />B. The Moon is covered with many craters.<br />C. The Moon has many bodies of liquid water.<br />D. The Moon has the ability to give off its own light.<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Which statement correctly describes a physical characteristic of the Moon?<br />A. The Moon is made of hot gases.<br />B. The Moon is covered with many craters.<br />C. The Moon has many bodies of liquid water.<br />D. The Moon has the ability to give off its own light.<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">The Moon is covered with many craters.</td>\n",
       "    <td class=\"gt_row gt_left\">The Moon is covered with many craters.</td>\n",
       "    <td class=\"gt_row gt_left\">The Moon is covered with many craters.</td>\n",
       "    <td class=\"gt_row gt_left\">B</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">a</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">The Moon has the ability to give off its own light</td>\n",
       "    <td class=\"gt_row gt_left\">D</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: qem</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 2</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">Which object in the solar system is orbited by a belt of asteroids?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Which object in the solar system is orbited by a belt of asteroids?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Which object in the solar system is orbited by a belt of asteroids?<br />A. Pluto<br />B. Saturn<br />C. the Sun<br />D. the Moon<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Which object in the solar system is orbited by a belt of asteroids?<br />A. Pluto<br />B. Saturn<br />C. the Sun<br />D. the Moon<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">the Sun</td>\n",
       "    <td class=\"gt_row gt_left\">the Sun</td>\n",
       "    <td class=\"gt_row gt_left\">the Sun</td>\n",
       "    <td class=\"gt_row gt_left\">C</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\"></td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">The Sun</td>\n",
       "    <td class=\"gt_row gt_left\">D</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: qem</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 3</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">Light waves that cross from an air medium to a water medium will</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Light waves that cross from an air medium to a water medium will<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Light waves that cross from an air medium to a water medium will<br />A. be focused into a straight line.<br />B. lose energy and dissipate.<br />C. change length and direction.<br />D. reflect off the water's surface.<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Light waves that cross from an air medium to a water medium will<br />A. be focused into a straight line.<br />B. lose energy and dissipate.<br />C. change length and direction.<br />D. reflect off the water's surface.<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">change length and direction.</td>\n",
       "    <td class=\"gt_row gt_left\">change length and direction.</td>\n",
       "    <td class=\"gt_row gt_left\">change length and direction.</td>\n",
       "    <td class=\"gt_row gt_left\">C</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">bend toward the normal</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">change length and direction</td>\n",
       "    <td class=\"gt_row gt_left\">D</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: qem</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 4</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">How many valence electrons does selenium have?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: How many valence electrons does selenium have?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: How many valence electrons does selenium have?<br />A. 3<br />B. 5<br />C. 6<br />D. 8<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: How many valence electrons does selenium have?<br />A. 3<br />B. 5<br />C. 6<br />D. 8<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">6</td>\n",
       "    <td class=\"gt_row gt_left\">6</td>\n",
       "    <td class=\"gt_row gt_left\">6</td>\n",
       "    <td class=\"gt_row gt_left\">C</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\"></td>\n",
       "    <td class=\"gt_row gt_left\">6</td>\n",
       "    <td class=\"gt_row gt_left\">6</td>\n",
       "    <td class=\"gt_row gt_left\">8</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: qem</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 5</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">As a warm moist air mass moving northward collides with a strong cold air mass moving southward, what observations will most likely be made?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: As a warm moist air mass moving northward collides with a strong cold air mass moving southward, what observations will most likely be made?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: As a warm moist air mass moving northward collides with a strong cold air mass moving southward, what observations will most likely be made?<br />A. Thick fog develops.<br />B. Temperatures increase.<br />C. Clouds begin to form.<br />D. Winds die down.<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: As a warm moist air mass moving northward collides with a strong cold air mass moving southward, what observations will most likely be made?<br />A. Thick fog develops.<br />B. Temperatures increase.<br />C. Clouds begin to form.<br />D. Winds die down.<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">Clouds begin to form.</td>\n",
       "    <td class=\"gt_row gt_left\">Clouds begin to form.</td>\n",
       "    <td class=\"gt_row gt_left\">Clouds begin to form.</td>\n",
       "    <td class=\"gt_row gt_left\">C</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">drought</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">Thick fog develops</td>\n",
       "    <td class=\"gt_row gt_left\">D</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: qem</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 6</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">Chemical weathering occurs when minerals in rocks are changed chemically. Which of these will most likely change the rate of chemical weathering on a rock?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Chemical weathering occurs when minerals in rocks are changed chemically. Which of these will most likely change the rate of chemical weathering on a rock?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Chemical weathering occurs when minerals in rocks are changed chemically. Which of these will most likely change the rate of chemical weathering on a rock?<br />A. decrease in air temperature<br />B. increase in rainfall amounts<br />C. slow movement of a glacier<br />D. rapid growth of plant roots<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Chemical weathering occurs when minerals in rocks are changed chemically. Which of these will most likely change the rate of chemical weathering on a rock?<br />A. decrease in air temperature<br />B. increase in rainfall amounts<br />C. slow movement of a glacier<br />D. rapid growth of plant roots<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">increase in rainfall amounts</td>\n",
       "    <td class=\"gt_row gt_left\">increase in rainfall amounts</td>\n",
       "    <td class=\"gt_row gt_left\">increase in rainfall amounts</td>\n",
       "    <td class=\"gt_row gt_left\">B</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">a</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">increase in rainfall amounts</td>\n",
       "    <td class=\"gt_row gt_left\">D</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: qem</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 7</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">A toy truck rolls over a smooth surface. If the surface is covered with sand, the truck will most likely roll</td>\n",
       "    <td class=\"gt_row gt_left\">Question: A toy truck rolls over a smooth surface. If the surface is covered with sand, the truck will most likely roll<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: A toy truck rolls over a smooth surface. If the surface is covered with sand, the truck will most likely roll<br />A. slower<br />B. faster<br />C. at the same speed<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: A toy truck rolls over a smooth surface. If the surface is covered with sand, the truck will most likely roll<br />A. slower<br />B. faster<br />C. at the same speed<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">slower</td>\n",
       "    <td class=\"gt_row gt_left\">slower</td>\n",
       "    <td class=\"gt_row gt_left\">slower</td>\n",
       "    <td class=\"gt_row gt_left\">A</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">farther</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">at the same speed</td>\n",
       "    <td class=\"gt_row gt_left\">C</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: qem</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 8</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">A town in northern Arkansas experienced colder than normal temperatures during part of the winter. Which change was most likely responsible for this?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: A town in northern Arkansas experienced colder than normal temperatures during part of the winter. Which change was most likely responsible for this?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: A town in northern Arkansas experienced colder than normal temperatures during part of the winter. Which change was most likely responsible for this?<br />A. A southward dip in the jet stream<br />B. A northward movement of the jet stream<br />C. The Coriolis effect creating a low pressure area<br />D. The Coriolis effect creating a high pressure area<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: A town in northern Arkansas experienced colder than normal temperatures during part of the winter. Which change was most likely responsible for this?<br />A. A southward dip in the jet stream<br />B. A northward movement of the jet stream<br />C. The Coriolis effect creating a low pressure area<br />D. The Coriolis effect creating a high pressure area<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">A southward dip in the jet stream</td>\n",
       "    <td class=\"gt_row gt_left\">A southward dip in the jet stream</td>\n",
       "    <td class=\"gt_row gt_left\">A southward dip in the jet stream</td>\n",
       "    <td class=\"gt_row gt_left\">A</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">The sun was farther away from the earth</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">The Coriolis effect creating a low pressure area</td>\n",
       "    <td class=\"gt_row gt_left\">D</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: qem</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 9</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">How much time is required for a bicycle to travel a distance of 100 m at an average speed of 2 m/s?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: How much time is required for a bicycle to travel a distance of 100 m at an average speed of 2 m/s?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: How much time is required for a bicycle to travel a distance of 100 m at an average speed of 2 m/s?<br />A. 0.02 s<br />B. 50 s<br />C. 100 s<br />D. 200 s<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: How much time is required for a bicycle to travel a distance of 100 m at an average speed of 2 m/s?<br />A. 0.02 s<br />B. 50 s<br />C. 100 s<br />D. 200 s<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">50 s</td>\n",
       "    <td class=\"gt_row gt_left\">50 s</td>\n",
       "    <td class=\"gt_row gt_left\">50 s</td>\n",
       "    <td class=\"gt_row gt_left\">B</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\"></td>\n",
       "    <td class=\"gt_row gt_left\">100 m/2 m/s = 50 s</td>\n",
       "    <td class=\"gt_row gt_left\">100 s</td>\n",
       "    <td class=\"gt_row gt_left\">200 s</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: qem</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "</tbody>\n",
       "\n",
       "\n",
       "</table>\n",
       "\n",
       "</div>\n",
       "        "
      ],
      "text/plain": [
       "GT(_tbl_data=shape: (40, 6)\n",
       "┌──────────┬──────────────┬──────────────────┬─────────────────┬─────────────────┬─────────────────┐\n",
       "│ Sample   ┆ Type         ┆ arc_base         ┆ arc_context     ┆ arc_context_cho ┆ arc_context_lab │\n",
       "│ ---      ┆ ---          ┆ ---              ┆ ---             ┆ ice             ┆ els             │\n",
       "│ str      ┆ str          ┆ str              ┆ str             ┆ ---             ┆ ---             │\n",
       "│          ┆              ┆                  ┆                 ┆ str             ┆ str             │\n",
       "╞══════════╪══════════════╪══════════════════╪═════════════════╪═════════════════╪═════════════════╡\n",
       "│ Sample 0 ┆ Example      ┆ Cities control   ┆ Question:       ┆ Question:       ┆ Question:       │\n",
       "│          ┆              ┆ the amount of p… ┆ Cities control  ┆ Cities control  ┆ Cities control  │\n",
       "│          ┆              ┆                  ┆ the a…          ┆ the a…          ┆ the a…          │\n",
       "│ Sample 0 ┆ Gold         ┆ The air stays    ┆ The air stays   ┆ The air stays   ┆ A               │\n",
       "│          ┆              ┆ cleaner.         ┆ cleaner.        ┆ cleaner.        ┆                 │\n",
       "│ Sample 0 ┆ Predictions  ┆ It reduces the   ┆ 1               ┆  It becomes     ┆  D              │\n",
       "│          ┆              ┆ amount of pollu… ┆                 ┆ safer to drive  ┆                 │\n",
       "│          ┆              ┆                  ┆                 ┆ on …            ┆                 │\n",
       "│ Sample 0 ┆ Metrics: qem ┆ 0                ┆ 0               ┆ 0               ┆ 0               │\n",
       "│ Sample 1 ┆ Example      ┆ Which statement  ┆ Question: Which ┆ Question: Which ┆ Question: Which │\n",
       "│          ┆              ┆ correctly desc…  ┆ statement corr… ┆ statement corr… ┆ statement corr… │\n",
       "│ …        ┆ …            ┆ …                ┆ …               ┆ …               ┆ …               │\n",
       "│ Sample 8 ┆ Metrics: qem ┆ 0                ┆ 0               ┆ 0               ┆ 0               │\n",
       "│ Sample 9 ┆ Example      ┆ How much time is ┆ Question: How   ┆ Question: How   ┆ Question: How   │\n",
       "│          ┆              ┆ required for …   ┆ much time is    ┆ much time is    ┆ much time is    │\n",
       "│          ┆              ┆                  ┆ req…            ┆ req…            ┆ req…            │\n",
       "│ Sample 9 ┆ Gold         ┆ 50 s             ┆ 50 s            ┆ 50 s            ┆ B               │\n",
       "│ Sample 9 ┆ Predictions  ┆                  ┆ 100 m/2 m/s =   ┆ 100 s           ┆ 200 s           │\n",
       "│          ┆              ┆                  ┆ 50 s            ┆                 ┆                 │\n",
       "│ Sample 9 ┆ Metrics: qem ┆ 0                ┆ 0               ┆ 0               ┆ 0               │\n",
       "└──────────┴──────────────┴──────────────────┴─────────────────┴─────────────────┴─────────────────┘, _body=<great_tables._gt_data.Body object at 0x7f074da26800>, _boxhead=Boxhead([ColInfo(var='Sample', type=<ColInfoTypeEnum.row_group: 3>, column_label='Sample', column_align='left', column_width=None), ColInfo(var='Type', type=<ColInfoTypeEnum.stub: 2>, column_label='Type', column_align='left', column_width=None), ColInfo(var='arc_base', type=<ColInfoTypeEnum.default: 1>, column_label='arc_base', column_align='left', column_width=None), ColInfo(var='arc_context', type=<ColInfoTypeEnum.default: 1>, column_label='arc_context', column_align='left', column_width=None), ColInfo(var='arc_context_choice', type=<ColInfoTypeEnum.default: 1>, column_label='arc_context_choice', column_align='left', column_width=None), ColInfo(var='arc_context_labels', type=<ColInfoTypeEnum.default: 1>, column_label='arc_context_labels', column_align='left', column_width=None)]), _stub=<great_tables._gt_data.Stub object at 0x7f074c181ea0>, _spanners=Spanners([SpannerInfo(spanner_id='Samples', spanner_level=0, spanner_label='Samples', spanner_units=None, spanner_pattern=None, vars=['arc_base', 'arc_context', 'arc_context_choice', 'arc_context_labels'], built=None)]), _heading=Heading(title=\"Comparing our different prompts' outputs\", subtitle=None, preheader=None), _stubhead=None, _source_notes=[], _footnotes=[], _styles=[], _locale=<great_tables._gt_data.Locale object at 0x7f074c1604f0>, _formats=[<great_tables._gt_data.FormatInfo object at 0x7f074c160e50>], _substitutions=[], _options=Options(table_id=OptionsInfo(scss=False, category='table', type='value', value=None), table_caption=OptionsInfo(scss=False, category='table', type='value', value=None), table_width=OptionsInfo(scss=True, category='table', type='px', value='auto'), table_layout=OptionsInfo(scss=True, category='table', type='value', value='fixed'), table_margin_left=OptionsInfo(scss=True, category='table', type='px', value='auto'), table_margin_right=OptionsInfo(scss=True, category='table', type='px', value='auto'), table_background_color=OptionsInfo(scss=True, category='table', type='value', value='#FFFFFF'), table_additional_css=OptionsInfo(scss=False, category='table', type='values', value=[]), table_font_names=OptionsInfo(scss=False, category='table', type='values', value=['-apple-system', 'BlinkMacSystemFont', 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarell', 'Helvetica Neue', 'Fira Sans', 'Droid Sans', 'Arial', 'sans-serif']), table_font_size=OptionsInfo(scss=True, category='table', type='px', value='16px'), table_font_weight=OptionsInfo(scss=True, category='table', type='value', value='normal'), table_font_style=OptionsInfo(scss=True, category='table', type='value', value='normal'), table_font_color=OptionsInfo(scss=True, category='table', type='value', value='#333333'), table_font_color_light=OptionsInfo(scss=True, category='table', type='value', value='#FFFFFF'), table_border_top_include=OptionsInfo(scss=False, category='table', type='boolean', value=True), table_border_top_style=OptionsInfo(scss=True, category='table', type='value', value='solid'), table_border_top_width=OptionsInfo(scss=True, category='table', type='px', value='2px'), table_border_top_color=OptionsInfo(scss=True, category='table', type='value', value='#A8A8A8'), table_border_right_style=OptionsInfo(scss=True, category='table', type='value', value='none'), table_border_right_width=OptionsInfo(scss=True, category='table', type='px', value='2px'), table_border_right_color=OptionsInfo(scss=True, category='table', type='value', value='#D3D3D3'), table_border_bottom_include=OptionsInfo(scss=False, category='table', type='boolean', value=True), table_border_bottom_style=OptionsInfo(scss=True, category='table', type='value', value='solid'), table_border_bottom_width=OptionsInfo(scss=True, category='table', type='px', value='2px'), table_border_bottom_color=OptionsInfo(scss=True, category='table', type='value', value='#A8A8A8'), table_border_left_style=OptionsInfo(scss=True, category='table', type='value', value='none'), table_border_left_width=OptionsInfo(scss=True, category='table', type='px', value='2px'), table_border_left_color=OptionsInfo(scss=True, category='table', type='value', value='#D3D3D3'), heading_background_color=OptionsInfo(scss=True, category='heading', type='value', value=None), heading_align=OptionsInfo(scss=True, category='heading', type='value', value='center'), heading_title_font_size=OptionsInfo(scss=True, category='heading', type='px', value='125%'), heading_title_font_weight=OptionsInfo(scss=True, category='heading', type='value', value='initial'), heading_subtitle_font_size=OptionsInfo(scss=True, category='heading', type='px', value='85%'), heading_subtitle_font_weight=OptionsInfo(scss=True, category='heading', type='value', value='initial'), heading_padding=OptionsInfo(scss=True, category='heading', type='px', value='4px'), heading_padding_horizontal=OptionsInfo(scss=True, category='heading', type='px', value='5px'), heading_border_bottom_style=OptionsInfo(scss=True, category='heading', type='value', value='solid'), heading_border_bottom_width=OptionsInfo(scss=True, category='heading', type='px', value='2px'), heading_border_bottom_color=OptionsInfo(scss=True, category='heading', type='value', value='#D3D3D3'), heading_border_lr_style=OptionsInfo(scss=True, category='heading', type='value', value='none'), heading_border_lr_width=OptionsInfo(scss=True, category='heading', type='px', value='1px'), heading_border_lr_color=OptionsInfo(scss=True, category='heading', type='value', value='#D3D3D3'), column_labels_background_color=OptionsInfo(scss=True, category='column_labels', type='value', value=None), column_labels_font_size=OptionsInfo(scss=True, category='column_labels', type='px', value='100%'), column_labels_font_weight=OptionsInfo(scss=True, category='column_labels', type='value', value='normal'), column_labels_text_transform=OptionsInfo(scss=True, category='column_labels', type='value', value='inherit'), column_labels_padding=OptionsInfo(scss=True, category='column_labels', type='px', value='5px'), column_labels_padding_horizontal=OptionsInfo(scss=True, category='column_labels', type='px', value='5px'), column_labels_vlines_style=OptionsInfo(scss=True, category='table_body', type='value', value='none'), column_labels_vlines_width=OptionsInfo(scss=True, category='table_body', type='px', value='1px'), column_labels_vlines_color=OptionsInfo(scss=True, category='table_body', type='value', value='#D3D3D3'), column_labels_border_top_style=OptionsInfo(scss=True, category='column_labels', type='value', value='solid'), column_labels_border_top_width=OptionsInfo(scss=True, category='column_labels', type='px', value='2px'), column_labels_border_top_color=OptionsInfo(scss=True, category='column_labels', type='value', value='#D3D3D3'), column_labels_border_bottom_style=OptionsInfo(scss=True, category='column_labels', type='value', value='solid'), column_labels_border_bottom_width=OptionsInfo(scss=True, category='column_labels', type='px', value='2px'), column_labels_border_bottom_color=OptionsInfo(scss=True, category='column_labels', type='value', value='#D3D3D3'), column_labels_border_lr_style=OptionsInfo(scss=True, category='column_labels', type='value', value='none'), column_labels_border_lr_width=OptionsInfo(scss=True, category='column_labels', type='px', value='1px'), column_labels_border_lr_color=OptionsInfo(scss=True, category='column_labels', type='value', value='#D3D3D3'), column_labels_hidden=OptionsInfo(scss=False, category='column_labels', type='boolean', value=False), row_group_background_color=OptionsInfo(scss=True, category='row_group', type='value', value=None), row_group_font_size=OptionsInfo(scss=True, category='row_group', type='px', value='100%'), row_group_font_weight=OptionsInfo(scss=True, category='row_group', type='value', value='initial'), row_group_text_transform=OptionsInfo(scss=True, category='row_group', type='value', value='inherit'), row_group_padding=OptionsInfo(scss=True, category='row_group', type='px', value='8px'), row_group_padding_horizontal=OptionsInfo(scss=True, category='row_group', type='px', value='5px'), row_group_border_top_style=OptionsInfo(scss=True, category='row_group', type='value', value='solid'), row_group_border_top_width=OptionsInfo(scss=True, category='row_group', type='px', value='2px'), row_group_border_top_color=OptionsInfo(scss=True, category='row_group', type='value', value='#D3D3D3'), row_group_border_right_style=OptionsInfo(scss=True, category='row_group', type='value', value='none'), row_group_border_right_width=OptionsInfo(scss=True, category='row_group', type='px', value='1px'), row_group_border_right_color=OptionsInfo(scss=True, category='row_group', type='value', value='#D3D3D3'), row_group_border_bottom_style=OptionsInfo(scss=True, category='row_group', type='value', value='solid'), row_group_border_bottom_width=OptionsInfo(scss=True, category='row_group', type='px', value='2px'), row_group_border_bottom_color=OptionsInfo(scss=True, category='row_group', type='value', value='#D3D3D3'), row_group_border_left_style=OptionsInfo(scss=True, category='row_group', type='value', value='none'), row_group_border_left_width=OptionsInfo(scss=True, category='row_group', type='px', value='1px'), row_group_border_left_color=OptionsInfo(scss=True, category='row_group', type='value', value='#D3D3D3'), row_group_as_column=OptionsInfo(scss=False, category='row_group', type='boolean', value=False), table_body_hlines_style=OptionsInfo(scss=True, category='table_body', type='value', value='solid'), table_body_hlines_width=OptionsInfo(scss=True, category='table_body', type='px', value='1px'), table_body_hlines_color=OptionsInfo(scss=True, category='table_body', type='value', value='#D3D3D3'), table_body_vlines_style=OptionsInfo(scss=True, category='table_body', type='value', value='none'), table_body_vlines_width=OptionsInfo(scss=True, category='table_body', type='px', value='1px'), table_body_vlines_color=OptionsInfo(scss=True, category='table_body', type='value', value='#D3D3D3'), table_body_border_top_style=OptionsInfo(scss=True, category='table_body', type='value', value='solid'), table_body_border_top_width=OptionsInfo(scss=True, category='table_body', type='px', value='2px'), table_body_border_top_color=OptionsInfo(scss=True, category='table_body', type='value', value='#D3D3D3'), table_body_border_bottom_style=OptionsInfo(scss=True, category='table_body', type='value', value='solid'), table_body_border_bottom_width=OptionsInfo(scss=True, category='table_body', type='px', value='2px'), table_body_border_bottom_color=OptionsInfo(scss=True, category='table_body', type='value', value='#D3D3D3'), data_row_padding=OptionsInfo(scss=True, category='data_row', type='px', value='8px'), data_row_padding_horizontal=OptionsInfo(scss=True, category='data_row', type='px', value='5px'), stub_background_color=OptionsInfo(scss=True, category='stub', type='value', value=None), stub_font_size=OptionsInfo(scss=True, category='stub', type='px', value='100%'), stub_font_weight=OptionsInfo(scss=True, category='stub', type='value', value='initial'), stub_text_transform=OptionsInfo(scss=True, category='stub', type='value', value='inherit'), stub_border_style=OptionsInfo(scss=True, category='stub', type='value', value='solid'), stub_border_width=OptionsInfo(scss=True, category='stub', type='px', value='2px'), stub_border_color=OptionsInfo(scss=True, category='stub', type='value', value='#D3D3D3'), stub_row_group_background_color=OptionsInfo(scss=True, category='stub', type='value', value=None), stub_row_group_font_size=OptionsInfo(scss=True, category='stub', type='px', value='100%'), stub_row_group_font_weight=OptionsInfo(scss=True, category='stub', type='value', value='initial'), stub_row_group_text_transform=OptionsInfo(scss=True, category='stub', type='value', value='inherit'), stub_row_group_border_style=OptionsInfo(scss=True, category='stub', type='value', value='solid'), stub_row_group_border_width=OptionsInfo(scss=True, category='stub', type='px', value='2px'), stub_row_group_border_color=OptionsInfo(scss=True, category='stub', type='value', value='#D3D3D3'), source_notes_padding=OptionsInfo(scss=True, category='source_notes', type='px', value='4px'), source_notes_padding_horizontal=OptionsInfo(scss=True, category='source_notes', type='px', value='5px'), source_notes_background_color=OptionsInfo(scss=True, category='source_notes', type='value', value=None), source_notes_font_size=OptionsInfo(scss=True, category='source_notes', type='px', value='90%'), source_notes_border_bottom_style=OptionsInfo(scss=True, category='source_notes', type='value', value='none'), source_notes_border_bottom_width=OptionsInfo(scss=True, category='source_notes', type='px', value='2px'), source_notes_border_bottom_color=OptionsInfo(scss=True, category='source_notes', type='value', value='#D3D3D3'), source_notes_border_lr_style=OptionsInfo(scss=True, category='source_notes', type='value', value='none'), source_notes_border_lr_width=OptionsInfo(scss=True, category='source_notes', type='px', value='2px'), source_notes_border_lr_color=OptionsInfo(scss=True, category='source_notes', type='value', value='#D3D3D3'), source_notes_multiline=OptionsInfo(scss=False, category='source_notes', type='boolean', value=True), source_notes_sep=OptionsInfo(scss=False, category='source_notes', type='value', value=' '), row_striping_background_color=OptionsInfo(scss=True, category='row', type='value', value='rgba(128,128,128,0.05)'), row_striping_include_stub=OptionsInfo(scss=False, category='row', type='boolean', value=False), row_striping_include_table_body=OptionsInfo(scss=False, category='row', type='boolean', value=False), container_width=OptionsInfo(scss=False, category='container', type='px', value='auto'), container_height=OptionsInfo(scss=False, category='container', type='px', value='auto'), container_padding_x=OptionsInfo(scss=False, category='container', type='px', value='0px'), container_padding_y=OptionsInfo(scss=False, category='container', type='px', value='10px'), container_overflow_x=OptionsInfo(scss=False, category='container', type='overflow', value='auto'), container_overflow_y=OptionsInfo(scss=False, category='container', type='overflow', value='auto'), quarto_disable_processing=OptionsInfo(scss=False, category='quarto', type='logical', value=False), quarto_use_bootstrap=OptionsInfo(scss=False, category='quarto', type='logical', value=False)), _has_built=False)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(GT(pl_data.head(max_samples*4))\n",
    "    .tab_header(\"Comparing our different prompts' outputs\")\n",
    "    .tab_spanner(label=\"Samples\", columns=cs.starts_with(\"arc\"))\n",
    "    .tab_stub(rowname_col=\"Type\", groupname_col=\"Sample\")\n",
    "     .fmt_markdown(columns=cs.starts_with(\"arc\"))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fda1fd6",
   "metadata": {},
   "source": [
    "可以发现：\n",
    "- 生成式评估对于基本格式过于困难：此时模型预测的结尾通常是错误的。\n",
    "- MCQA 评估对于基本 + 问答标签格式也不容易：此时模型输出通常为数字。\n",
    "- 问题中引入选项有助于模型输出选项中的一个。\n",
    "\n",
    "在最后一种情况下，当问题中存在选项但模型必须预测标签时，可能会出现预测错误。\n",
    "\n",
    "还有一些其他情况 (如样本 3 和 4)，模型在预测标签与预测选项 (如果选项存在) 时，输出可能不一致。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c2fe99d",
   "metadata": {},
   "source": [
    "### 来看一下对数概率输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "1c5b7411",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"zbompcphoq\" style=\"padding-left:0px;padding-right:0px;padding-top:10px;padding-bottom:10px;overflow-x:auto;overflow-y:auto;width:auto;height:auto;\">\n",
       "<style>\n",
       "#zbompcphoq table {\n",
       "          font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;\n",
       "          -webkit-font-smoothing: antialiased;\n",
       "          -moz-osx-font-smoothing: grayscale;\n",
       "        }\n",
       "\n",
       "#zbompcphoq thead, tbody, tfoot, tr, td, th { border-style: none; }\n",
       " tr { background-color: transparent; }\n",
       "#zbompcphoq p { margin: 0; padding: 0; }\n",
       " #zbompcphoq .gt_table { display: table; border-collapse: collapse; line-height: normal; margin-left: auto; margin-right: auto; color: #333333; font-size: 16px; font-weight: normal; font-style: normal; background-color: #FFFFFF; width: auto; border-top-style: solid; border-top-width: 2px; border-top-color: #A8A8A8; border-right-style: none; border-right-width: 2px; border-right-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #A8A8A8; border-left-style: none; border-left-width: 2px; border-left-color: #D3D3D3; }\n",
       " #zbompcphoq .gt_caption { padding-top: 4px; padding-bottom: 4px; }\n",
       " #zbompcphoq .gt_title { color: #333333; font-size: 125%; font-weight: initial; padding-top: 4px; padding-bottom: 4px; padding-left: 5px; padding-right: 5px; border-bottom-color: #FFFFFF; border-bottom-width: 0; }\n",
       " #zbompcphoq .gt_subtitle { color: #333333; font-size: 85%; font-weight: initial; padding-top: 3px; padding-bottom: 5px; padding-left: 5px; padding-right: 5px; border-top-color: #FFFFFF; border-top-width: 0; }\n",
       " #zbompcphoq .gt_heading { background-color: #FFFFFF; text-align: center; border-bottom-color: #FFFFFF; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; }\n",
       " #zbompcphoq .gt_bottom_border { border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; }\n",
       " #zbompcphoq .gt_col_headings { border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; }\n",
       " #zbompcphoq .gt_col_heading { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: normal; text-transform: inherit; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; vertical-align: bottom; padding-top: 5px; padding-bottom: 5px; padding-left: 5px; padding-right: 5px; overflow-x: hidden; }\n",
       " #zbompcphoq .gt_column_spanner_outer { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: normal; text-transform: inherit; padding-top: 0; padding-bottom: 0; padding-left: 4px; padding-right: 4px; }\n",
       " #zbompcphoq .gt_column_spanner_outer:first-child { padding-left: 0; }\n",
       " #zbompcphoq .gt_column_spanner_outer:last-child { padding-right: 0; }\n",
       " #zbompcphoq .gt_column_spanner { border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; vertical-align: bottom; padding-top: 5px; padding-bottom: 5px; overflow-x: hidden; display: inline-block; width: 100%; }\n",
       " #zbompcphoq .gt_spanner_row { border-bottom-style: hidden; }\n",
       " #zbompcphoq .gt_group_heading { padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px; color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; text-transform: inherit; border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; vertical-align: middle; text-align: left; }\n",
       " #zbompcphoq .gt_empty_group_heading { padding: 0.5px; color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; vertical-align: middle; }\n",
       " #zbompcphoq .gt_from_md> :first-child { margin-top: 0; }\n",
       " #zbompcphoq .gt_from_md> :last-child { margin-bottom: 0; }\n",
       " #zbompcphoq .gt_row { padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px; margin: 10px; border-top-style: solid; border-top-width: 1px; border-top-color: #D3D3D3; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; vertical-align: middle; overflow-x: hidden; }\n",
       " #zbompcphoq .gt_stub { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; text-transform: inherit; border-right-style: solid; border-right-width: 2px; border-right-color: #D3D3D3; padding-left: 5px; padding-right: 5px; }\n",
       " #zbompcphoq .gt_stub_row_group { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; text-transform: inherit; border-right-style: solid; border-right-width: 2px; border-right-color: #D3D3D3; padding-left: 5px; padding-right: 5px; vertical-align: top; }\n",
       " #zbompcphoq .gt_row_group_first td { border-top-width: 2px; }\n",
       " #zbompcphoq .gt_row_group_first th { border-top-width: 2px; }\n",
       " #zbompcphoq .gt_striped { background-color: rgba(128,128,128,0.05); }\n",
       " #zbompcphoq .gt_table_body { border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; }\n",
       " #zbompcphoq .gt_sourcenotes { color: #333333; background-color: #FFFFFF; border-bottom-style: none; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 2px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 2px; border-right-color: #D3D3D3; }\n",
       " #zbompcphoq .gt_sourcenote { font-size: 90%; padding-top: 4px; padding-bottom: 4px; padding-left: 5px; padding-right: 5px; text-align: left; }\n",
       " #zbompcphoq .gt_left { text-align: left; }\n",
       " #zbompcphoq .gt_center { text-align: center; }\n",
       " #zbompcphoq .gt_right { text-align: right; font-variant-numeric: tabular-nums; }\n",
       " #zbompcphoq .gt_font_normal { font-weight: normal; }\n",
       " #zbompcphoq .gt_font_bold { font-weight: bold; }\n",
       " #zbompcphoq .gt_font_italic { font-style: italic; }\n",
       " #zbompcphoq .gt_super { font-size: 65%; }\n",
       " #zbompcphoq .gt_footnote_marks { font-size: 75%; vertical-align: 0.4em; position: initial; }\n",
       " #zbompcphoq .gt_asterisk { font-size: 100%; vertical-align: 0; }\n",
       " \n",
       "</style>\n",
       "<table class=\"gt_table\" data-quarto-disable-processing=\"false\" data-quarto-bootstrap=\"false\">\n",
       "<thead>\n",
       "\n",
       "  <tr class=\"gt_heading\">\n",
       "    <td colspan=\"5\" class=\"gt_heading gt_title gt_font_normal\">Comparing our different prompts&#x27; outputs</td>\n",
       "  </tr>\n",
       "<tr class=\"gt_col_headings gt_spanner_row\">\n",
       "  <th class=\"gt_col_heading gt_columns_bottom_border gt_left\" rowspan=\"2\" colspan=\"1\" scope=\"col\" id=\"\"></th>\n",
       "  <th class=\"gt_center gt_columns_top_border gt_column_spanner_outer\" rowspan=\"1\" colspan=\"4\" scope=\"colgroup\" id=\"Samples\">\n",
       "    <span class=\"gt_column_spanner\">Samples</span>\n",
       "  </th>\n",
       "</tr>\n",
       "<tr class=\"gt_col_headings\">\n",
       "  <th class=\"gt_col_heading gt_columns_bottom_border gt_left\" rowspan=\"1\" colspan=\"1\" scope=\"col\" id=\"arc_base\">arc_base</th>\n",
       "  <th class=\"gt_col_heading gt_columns_bottom_border gt_left\" rowspan=\"1\" colspan=\"1\" scope=\"col\" id=\"arc_context\">arc_context</th>\n",
       "  <th class=\"gt_col_heading gt_columns_bottom_border gt_left\" rowspan=\"1\" colspan=\"1\" scope=\"col\" id=\"arc_context_choice\">arc_context_choice</th>\n",
       "  <th class=\"gt_col_heading gt_columns_bottom_border gt_left\" rowspan=\"1\" colspan=\"1\" scope=\"col\" id=\"arc_context_labels\">arc_context_labels</th>\n",
       "</tr>\n",
       "</thead>\n",
       "<tbody class=\"gt_table_body\">\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 10</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">Cities control the amount of pollution that is allowed to come from cars. How does this most likely help people?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Cities control the amount of pollution that is allowed to come from cars. How does this most likely help people?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Cities control the amount of pollution that is allowed to come from cars. How does this most likely help people?<br />A. The air stays cleaner.<br />B. Cars can travel at faster speeds.<br />C. The skills of the drivers improve.<br />D. It becomes safer to drive on the roads.<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Cities control the amount of pollution that is allowed to come from cars. How does this most likely help people?<br />A. The air stays cleaner.<br />B. Cars can travel at faster speeds.<br />C. The skills of the drivers improve.<br />D. It becomes safer to drive on the roads.<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">The air stays cleaner.</td>\n",
       "    <td class=\"gt_row gt_left\">The air stays cleaner.</td>\n",
       "    <td class=\"gt_row gt_left\">The air stays cleaner.</td>\n",
       "    <td class=\"gt_row gt_left\">A</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">(-10.595719337463379, False)<br />(-17.943925857543945, False)<br />(-25.558387756347656, False)<br />(-19.944787979125977, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-9.650793075561523, False)<br />(-18.44221305847168, False)<br />(-28.147789001464844, False)<br />(-21.082042694091797, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-1.7918881177902222, False)<br />(-3.5758469104766846, False)<br />(-2.042778253555298, False)<br />(-0.5759693384170532, True)</td>\n",
       "    <td class=\"gt_row gt_left\">(-1.445155382156372, False)<br />(-1.695155382156372, False)<br />(-1.445155382156372, False)<br />(-1.070155382156372, True)</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: acc_norm</th>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 11</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">Which statement correctly describes a physical characteristic of the Moon?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Which statement correctly describes a physical characteristic of the Moon?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Which statement correctly describes a physical characteristic of the Moon?<br />A. The Moon is made of hot gases.<br />B. The Moon is covered with many craters.<br />C. The Moon has many bodies of liquid water.<br />D. The Moon has the ability to give off its own light.<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Which statement correctly describes a physical characteristic of the Moon?<br />A. The Moon is made of hot gases.<br />B. The Moon is covered with many craters.<br />C. The Moon has many bodies of liquid water.<br />D. The Moon has the ability to give off its own light.<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">The Moon is covered with many craters.</td>\n",
       "    <td class=\"gt_row gt_left\">The Moon is covered with many craters.</td>\n",
       "    <td class=\"gt_row gt_left\">The Moon is covered with many craters.</td>\n",
       "    <td class=\"gt_row gt_left\">B</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">(-14.036760330200195, False)<br />(-14.156133651733398, False)<br />(-23.221176147460938, False)<br />(-22.99703598022461, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-12.965611457824707, False)<br />(-11.864521026611328, False)<br />(-20.491010665893555, False)<br />(-21.728534698486328, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-4.868813514709473, False)<br />(-3.7199177742004395, False)<br />(-3.739739418029785, False)<br />(-0.2140919268131256, True)</td>\n",
       "    <td class=\"gt_row gt_left\">(-1.5505614280700684, False)<br />(-1.4255614280700684, False)<br />(-1.3005614280700684, True)<br />(-1.3005614280700684, False)</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: acc_norm</th>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 12</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">Which object in the solar system is orbited by a belt of asteroids?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Which object in the solar system is orbited by a belt of asteroids?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Which object in the solar system is orbited by a belt of asteroids?<br />A. Pluto<br />B. Saturn<br />C. the Sun<br />D. the Moon<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Which object in the solar system is orbited by a belt of asteroids?<br />A. Pluto<br />B. Saturn<br />C. the Sun<br />D. the Moon<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">the Sun</td>\n",
       "    <td class=\"gt_row gt_left\">the Sun</td>\n",
       "    <td class=\"gt_row gt_left\">the Sun</td>\n",
       "    <td class=\"gt_row gt_left\">C</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">(-5.751784801483154, False)<br />(-3.7467873096466064, False)<br />(-6.4582719802856445, False)<br />(-7.0832719802856445, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-3.279644250869751, False)<br />(-3.654644250869751, False)<br />(-5.170290946960449, False)<br />(-5.920290946960449, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-2.128627300262451, False)<br />(-1.8786273002624512, False)<br />(-0.7749634981155396, True)<br />(-2.774963617324829, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-1.6950974464416504, False)<br />(-1.4450974464416504, False)<br />(-1.3200974464416504, False)<br />(-1.1950974464416504, True)</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: acc_norm</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 13</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">Light waves that cross from an air medium to a water medium will</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Light waves that cross from an air medium to a water medium will<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Light waves that cross from an air medium to a water medium will<br />A. be focused into a straight line.<br />B. lose energy and dissipate.<br />C. change length and direction.<br />D. reflect off the water's surface.<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Light waves that cross from an air medium to a water medium will<br />A. be focused into a straight line.<br />B. lose energy and dissipate.<br />C. change length and direction.<br />D. reflect off the water's surface.<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">change length and direction.</td>\n",
       "    <td class=\"gt_row gt_left\">change length and direction.</td>\n",
       "    <td class=\"gt_row gt_left\">change length and direction.</td>\n",
       "    <td class=\"gt_row gt_left\">C</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">(-25.314556121826172, False)<br />(-22.782466888427734, False)<br />(-22.68366050720215, False)<br />(-21.23833465576172, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-17.779306411743164, False)<br />(-15.633138656616211, False)<br />(-14.039811134338379, False)<br />(-11.432372093200684, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-4.998326301574707, False)<br />(-1.623903512954712, False)<br />(-0.5861192941665649, True)<br />(-2.0934457778930664, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-1.625598669052124, False)<br />(-1.500598669052124, False)<br />(-1.375598669052124, False)<br />(-1.125598669052124, True)</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: acc_norm</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 14</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">How many valence electrons does selenium have?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: How many valence electrons does selenium have?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: How many valence electrons does selenium have?<br />A. 3<br />B. 5<br />C. 6<br />D. 8<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: How many valence electrons does selenium have?<br />A. 3<br />B. 5<br />C. 6<br />D. 8<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">6</td>\n",
       "    <td class=\"gt_row gt_left\">6</td>\n",
       "    <td class=\"gt_row gt_left\">6</td>\n",
       "    <td class=\"gt_row gt_left\">C</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">(-6.6732683181762695, False)<br />(-6.9232683181762695, False)<br />(-7.2357683181762695, False)<br />(-6.7982683181762695, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-3.4133384227752686, False)<br />(-4.663338661193848, False)<br />(-1.1633384227752686, True)<br />(-3.7883384227752686, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-6.926815986633301, False)<br />(-6.051815986633301, False)<br />(-0.17681613564491272, True)<br />(-1.9268162250518799, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-1.8513681888580322, False)<br />(-1.6013681888580322, False)<br />(-1.3513681888580322, False)<br />(-0.976368248462677, True)</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: acc_norm</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 15</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">As a warm moist air mass moving northward collides with a strong cold air mass moving southward, what observations will most likely be made?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: As a warm moist air mass moving northward collides with a strong cold air mass moving southward, what observations will most likely be made?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: As a warm moist air mass moving northward collides with a strong cold air mass moving southward, what observations will most likely be made?<br />A. Thick fog develops.<br />B. Temperatures increase.<br />C. Clouds begin to form.<br />D. Winds die down.<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: As a warm moist air mass moving northward collides with a strong cold air mass moving southward, what observations will most likely be made?<br />A. Thick fog develops.<br />B. Temperatures increase.<br />C. Clouds begin to form.<br />D. Winds die down.<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">Clouds begin to form.</td>\n",
       "    <td class=\"gt_row gt_left\">Clouds begin to form.</td>\n",
       "    <td class=\"gt_row gt_left\">Clouds begin to form.</td>\n",
       "    <td class=\"gt_row gt_left\">C</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">(-16.48362159729004, False)<br />(-12.333986282348633, False)<br />(-10.768516540527344, False)<br />(-14.52383804321289, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-15.831233024597168, False)<br />(-11.781991004943848, False)<br />(-10.162830352783203, False)<br />(-15.114853858947754, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-0.6979869604110718, True)<br />(-1.7184417247772217, False)<br />(-2.8269424438476562, False)<br />(-2.2389371395111084, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-1.4881373643875122, False)<br />(-1.4881373643875122, False)<br />(-1.3631373643875122, False)<br />(-1.2381373643875122, True)</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: acc_norm</th>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 16</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">Chemical weathering occurs when minerals in rocks are changed chemically. Which of these will most likely change the rate of chemical weathering on a rock?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Chemical weathering occurs when minerals in rocks are changed chemically. Which of these will most likely change the rate of chemical weathering on a rock?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Chemical weathering occurs when minerals in rocks are changed chemically. Which of these will most likely change the rate of chemical weathering on a rock?<br />A. decrease in air temperature<br />B. increase in rainfall amounts<br />C. slow movement of a glacier<br />D. rapid growth of plant roots<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: Chemical weathering occurs when minerals in rocks are changed chemically. Which of these will most likely change the rate of chemical weathering on a rock?<br />A. decrease in air temperature<br />B. increase in rainfall amounts<br />C. slow movement of a glacier<br />D. rapid growth of plant roots<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">increase in rainfall amounts</td>\n",
       "    <td class=\"gt_row gt_left\">increase in rainfall amounts</td>\n",
       "    <td class=\"gt_row gt_left\">increase in rainfall amounts</td>\n",
       "    <td class=\"gt_row gt_left\">B</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">(-15.226205825805664, False)<br />(-18.300310134887695, False)<br />(-18.7595272064209, False)<br />(-16.90313148498535, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-15.182734489440918, False)<br />(-16.29802131652832, False)<br />(-15.929485321044922, False)<br />(-15.493678092956543, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-2.301724433898926, False)<br />(-1.6302365064620972, False)<br />(-1.4298421144485474, False)<br />(-2.3473901748657227, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-1.734103798866272, False)<br />(-1.609103798866272, False)<br />(-1.234103798866272, False)<br />(-1.109103798866272, True)</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: acc_norm</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 17</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">A toy truck rolls over a smooth surface. If the surface is covered with sand, the truck will most likely roll</td>\n",
       "    <td class=\"gt_row gt_left\">Question: A toy truck rolls over a smooth surface. If the surface is covered with sand, the truck will most likely roll<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: A toy truck rolls over a smooth surface. If the surface is covered with sand, the truck will most likely roll<br />A. slower<br />B. faster<br />C. at the same speed<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: A toy truck rolls over a smooth surface. If the surface is covered with sand, the truck will most likely roll<br />A. slower<br />B. faster<br />C. at the same speed<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">slower</td>\n",
       "    <td class=\"gt_row gt_left\">slower</td>\n",
       "    <td class=\"gt_row gt_left\">slower</td>\n",
       "    <td class=\"gt_row gt_left\">A</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">(-13.275317192077637, False)<br />(-10.096747398376465, False)<br />(-19.148441314697266, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-2.1855695247650146, False)<br />(-2.1855695247650146, True)<br />(-7.597217559814453, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-1.2695996761322021, False)<br />(-0.7695997357368469, True)<br />(-1.6472738981246948, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-1.2452011108398438, False)<br />(-1.3702011108398438, False)<br />(-0.995201051235199, True)</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: acc_norm</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">1</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 18</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">A town in northern Arkansas experienced colder than normal temperatures during part of the winter. Which change was most likely responsible for this?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: A town in northern Arkansas experienced colder than normal temperatures during part of the winter. Which change was most likely responsible for this?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: A town in northern Arkansas experienced colder than normal temperatures during part of the winter. Which change was most likely responsible for this?<br />A. A southward dip in the jet stream<br />B. A northward movement of the jet stream<br />C. The Coriolis effect creating a low pressure area<br />D. The Coriolis effect creating a high pressure area<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: A town in northern Arkansas experienced colder than normal temperatures during part of the winter. Which change was most likely responsible for this?<br />A. A southward dip in the jet stream<br />B. A northward movement of the jet stream<br />C. The Coriolis effect creating a low pressure area<br />D. The Coriolis effect creating a high pressure area<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">A southward dip in the jet stream</td>\n",
       "    <td class=\"gt_row gt_left\">A southward dip in the jet stream</td>\n",
       "    <td class=\"gt_row gt_left\">A southward dip in the jet stream</td>\n",
       "    <td class=\"gt_row gt_left\">A</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">(-14.345656394958496, False)<br />(-16.05936050415039, False)<br />(-28.54291534423828, False)<br />(-28.780471801757812, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-14.603858947753906, False)<br />(-16.18315887451172, False)<br />(-27.121496200561523, False)<br />(-27.188758850097656, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-1.6069732904434204, False)<br />(-3.61207914352417, False)<br />(-0.9148926734924316, True)<br />(-1.2960143089294434, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-1.4954701662063599, False)<br />(-1.6204701662063599, False)<br />(-1.2454701662063599, True)<br />(-1.2454701662063599, False)</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: acc_norm</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "  <tr class=\"gt_group_heading_row\">\n",
       "    <th class=\"gt_group_heading\" colspan=\"5\">Sample 19</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Example</th>\n",
       "    <td class=\"gt_row gt_left\">How much time is required for a bicycle to travel a distance of 100 m at an average speed of 2 m/s?</td>\n",
       "    <td class=\"gt_row gt_left\">Question: How much time is required for a bicycle to travel a distance of 100 m at an average speed of 2 m/s?<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: How much time is required for a bicycle to travel a distance of 100 m at an average speed of 2 m/s?<br />A. 0.02 s<br />B. 50 s<br />C. 100 s<br />D. 200 s<br />Answer:</td>\n",
       "    <td class=\"gt_row gt_left\">Question: How much time is required for a bicycle to travel a distance of 100 m at an average speed of 2 m/s?<br />A. 0.02 s<br />B. 50 s<br />C. 100 s<br />D. 200 s<br />Answer:</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Gold</th>\n",
       "    <td class=\"gt_row gt_left\">50 s</td>\n",
       "    <td class=\"gt_row gt_left\">50 s</td>\n",
       "    <td class=\"gt_row gt_left\">50 s</td>\n",
       "    <td class=\"gt_row gt_left\">B</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Predictions</th>\n",
       "    <td class=\"gt_row gt_left\">(-9.211626052856445, False)<br />(-7.683445453643799, False)<br />(-7.148634433746338, False)<br />(-8.429720878601074, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-7.6334710121154785, False)<br />(-4.587807655334473, False)<br />(-4.615830421447754, False)<br />(-6.059815883636475, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-3.2665140628814697, False)<br />(-1.9924125671386719, False)<br />(-0.8778499960899353, True)<br />(-0.9978684186935425, False)</td>\n",
       "    <td class=\"gt_row gt_left\">(-1.7351858615875244, False)<br />(-1.6101858615875244, False)<br />(-1.2351858615875244, False)<br />(-1.1101858615875244, True)</td>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th class=\"gt_row gt_left gt_stub\">Metrics: acc_norm</th>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "    <td class=\"gt_row gt_left\">0</td>\n",
       "  </tr>\n",
       "</tbody>\n",
       "\n",
       "\n",
       "</table>\n",
       "\n",
       "</div>\n",
       "        "
      ],
      "text/plain": [
       "GT(_tbl_data=shape: (40, 6)\n",
       "┌───────────┬─────────────┬──────────────────┬─────────────────┬─────────────────┬─────────────────┐\n",
       "│ Sample    ┆ Type        ┆ arc_base         ┆ arc_context     ┆ arc_context_cho ┆ arc_context_lab │\n",
       "│ ---       ┆ ---         ┆ ---              ┆ ---             ┆ ice             ┆ els             │\n",
       "│ str       ┆ str         ┆ str              ┆ str             ┆ ---             ┆ ---             │\n",
       "│           ┆             ┆                  ┆                 ┆ str             ┆ str             │\n",
       "╞═══════════╪═════════════╪══════════════════╪═════════════════╪═════════════════╪═════════════════╡\n",
       "│ Sample 10 ┆ Example     ┆ Cities control   ┆ Question:       ┆ Question:       ┆ Question:       │\n",
       "│           ┆             ┆ the amount of p… ┆ Cities control  ┆ Cities control  ┆ Cities control  │\n",
       "│           ┆             ┆                  ┆ the a…          ┆ the a…          ┆ the a…          │\n",
       "│ Sample 10 ┆ Gold        ┆ The air stays    ┆ The air stays   ┆ The air stays   ┆ A               │\n",
       "│           ┆             ┆ cleaner.         ┆ cleaner.        ┆ cleaner.        ┆                 │\n",
       "│ Sample 10 ┆ Predictions ┆ (-10.59571933746 ┆ (-9.65079307556 ┆ (-1.79188811779 ┆ (-1.44515538215 │\n",
       "│           ┆             ┆ 3379, False)<b…  ┆ 1523,           ┆ 02222,          ┆ 6372,           │\n",
       "│           ┆             ┆                  ┆ False)<br…      ┆ False)<b…       ┆ False)<br…      │\n",
       "│ Sample 10 ┆ Metrics:    ┆ 1                ┆ 1               ┆ 0               ┆ 0               │\n",
       "│           ┆ acc_norm    ┆                  ┆                 ┆                 ┆                 │\n",
       "│ Sample 11 ┆ Example     ┆ Which statement  ┆ Question: Which ┆ Question: Which ┆ Question: Which │\n",
       "│           ┆             ┆ correctly desc…  ┆ statement corr… ┆ statement corr… ┆ statement corr… │\n",
       "│ …         ┆ …           ┆ …                ┆ …               ┆ …               ┆ …               │\n",
       "│ Sample 18 ┆ Metrics:    ┆ 0                ┆ 0               ┆ 0               ┆ 0               │\n",
       "│           ┆ acc_norm    ┆                  ┆                 ┆                 ┆                 │\n",
       "│ Sample 19 ┆ Example     ┆ How much time is ┆ Question: How   ┆ Question: How   ┆ Question: How   │\n",
       "│           ┆             ┆ required for …   ┆ much time is    ┆ much time is    ┆ much time is    │\n",
       "│           ┆             ┆                  ┆ req…            ┆ req…            ┆ req…            │\n",
       "│ Sample 19 ┆ Gold        ┆ 50 s             ┆ 50 s            ┆ 50 s            ┆ B               │\n",
       "│ Sample 19 ┆ Predictions ┆ (-9.211626052856 ┆ (-7.63347101211 ┆ (-3.26651406288 ┆ (-1.73518586158 │\n",
       "│           ┆             ┆ 445, False)<br…  ┆ 54785,          ┆ 14697,          ┆ 75244,          │\n",
       "│           ┆             ┆                  ┆ False)<b…       ┆ False)<b…       ┆ False)<b…       │\n",
       "│ Sample 19 ┆ Metrics:    ┆ 0                ┆ 0               ┆ 0               ┆ 0               │\n",
       "│           ┆ acc_norm    ┆                  ┆                 ┆                 ┆                 │\n",
       "└───────────┴─────────────┴──────────────────┴─────────────────┴─────────────────┴─────────────────┘, _body=<great_tables._gt_data.Body object at 0x7f074c8b4a30>, _boxhead=Boxhead([ColInfo(var='Sample', type=<ColInfoTypeEnum.row_group: 3>, column_label='Sample', column_align='left', column_width=None), ColInfo(var='Type', type=<ColInfoTypeEnum.stub: 2>, column_label='Type', column_align='left', column_width=None), ColInfo(var='arc_base', type=<ColInfoTypeEnum.default: 1>, column_label='arc_base', column_align='left', column_width=None), ColInfo(var='arc_context', type=<ColInfoTypeEnum.default: 1>, column_label='arc_context', column_align='left', column_width=None), ColInfo(var='arc_context_choice', type=<ColInfoTypeEnum.default: 1>, column_label='arc_context_choice', column_align='left', column_width=None), ColInfo(var='arc_context_labels', type=<ColInfoTypeEnum.default: 1>, column_label='arc_context_labels', column_align='left', column_width=None)]), _stub=<great_tables._gt_data.Stub object at 0x7f074c12a890>, _spanners=Spanners([SpannerInfo(spanner_id='Samples', spanner_level=0, spanner_label='Samples', spanner_units=None, spanner_pattern=None, vars=['arc_base', 'arc_context', 'arc_context_choice', 'arc_context_labels'], built=None)]), _heading=Heading(title=\"Comparing our different prompts' outputs\", subtitle=None, preheader=None), _stubhead=None, _source_notes=[], _footnotes=[], _styles=[], _locale=<great_tables._gt_data.Locale object at 0x7f074c8b6170>, _formats=[<great_tables._gt_data.FormatInfo object at 0x7f074c8c7880>], _substitutions=[], _options=Options(table_id=OptionsInfo(scss=False, category='table', type='value', value=None), table_caption=OptionsInfo(scss=False, category='table', type='value', value=None), table_width=OptionsInfo(scss=True, category='table', type='px', value='auto'), table_layout=OptionsInfo(scss=True, category='table', type='value', value='fixed'), table_margin_left=OptionsInfo(scss=True, category='table', type='px', value='auto'), table_margin_right=OptionsInfo(scss=True, category='table', type='px', value='auto'), table_background_color=OptionsInfo(scss=True, category='table', type='value', value='#FFFFFF'), table_additional_css=OptionsInfo(scss=False, category='table', type='values', value=[]), table_font_names=OptionsInfo(scss=False, category='table', type='values', value=['-apple-system', 'BlinkMacSystemFont', 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarell', 'Helvetica Neue', 'Fira Sans', 'Droid Sans', 'Arial', 'sans-serif']), table_font_size=OptionsInfo(scss=True, category='table', type='px', value='16px'), table_font_weight=OptionsInfo(scss=True, category='table', type='value', value='normal'), table_font_style=OptionsInfo(scss=True, category='table', type='value', value='normal'), table_font_color=OptionsInfo(scss=True, category='table', type='value', value='#333333'), table_font_color_light=OptionsInfo(scss=True, category='table', type='value', value='#FFFFFF'), table_border_top_include=OptionsInfo(scss=False, category='table', type='boolean', value=True), table_border_top_style=OptionsInfo(scss=True, category='table', type='value', value='solid'), table_border_top_width=OptionsInfo(scss=True, category='table', type='px', value='2px'), table_border_top_color=OptionsInfo(scss=True, category='table', type='value', value='#A8A8A8'), table_border_right_style=OptionsInfo(scss=True, category='table', type='value', value='none'), table_border_right_width=OptionsInfo(scss=True, category='table', type='px', value='2px'), table_border_right_color=OptionsInfo(scss=True, category='table', type='value', value='#D3D3D3'), table_border_bottom_include=OptionsInfo(scss=False, category='table', type='boolean', value=True), table_border_bottom_style=OptionsInfo(scss=True, category='table', type='value', value='solid'), table_border_bottom_width=OptionsInfo(scss=True, category='table', type='px', value='2px'), table_border_bottom_color=OptionsInfo(scss=True, category='table', type='value', value='#A8A8A8'), table_border_left_style=OptionsInfo(scss=True, category='table', type='value', value='none'), table_border_left_width=OptionsInfo(scss=True, category='table', type='px', value='2px'), table_border_left_color=OptionsInfo(scss=True, category='table', type='value', value='#D3D3D3'), heading_background_color=OptionsInfo(scss=True, category='heading', type='value', value=None), heading_align=OptionsInfo(scss=True, category='heading', type='value', value='center'), heading_title_font_size=OptionsInfo(scss=True, category='heading', type='px', value='125%'), heading_title_font_weight=OptionsInfo(scss=True, category='heading', type='value', value='initial'), heading_subtitle_font_size=OptionsInfo(scss=True, category='heading', type='px', value='85%'), heading_subtitle_font_weight=OptionsInfo(scss=True, category='heading', type='value', value='initial'), heading_padding=OptionsInfo(scss=True, category='heading', type='px', value='4px'), heading_padding_horizontal=OptionsInfo(scss=True, category='heading', type='px', value='5px'), heading_border_bottom_style=OptionsInfo(scss=True, category='heading', type='value', value='solid'), heading_border_bottom_width=OptionsInfo(scss=True, category='heading', type='px', value='2px'), heading_border_bottom_color=OptionsInfo(scss=True, category='heading', type='value', value='#D3D3D3'), heading_border_lr_style=OptionsInfo(scss=True, category='heading', type='value', value='none'), heading_border_lr_width=OptionsInfo(scss=True, category='heading', type='px', value='1px'), heading_border_lr_color=OptionsInfo(scss=True, category='heading', type='value', value='#D3D3D3'), column_labels_background_color=OptionsInfo(scss=True, category='column_labels', type='value', value=None), column_labels_font_size=OptionsInfo(scss=True, category='column_labels', type='px', value='100%'), column_labels_font_weight=OptionsInfo(scss=True, category='column_labels', type='value', value='normal'), column_labels_text_transform=OptionsInfo(scss=True, category='column_labels', type='value', value='inherit'), column_labels_padding=OptionsInfo(scss=True, category='column_labels', type='px', value='5px'), column_labels_padding_horizontal=OptionsInfo(scss=True, category='column_labels', type='px', value='5px'), column_labels_vlines_style=OptionsInfo(scss=True, category='table_body', type='value', value='none'), column_labels_vlines_width=OptionsInfo(scss=True, category='table_body', type='px', value='1px'), column_labels_vlines_color=OptionsInfo(scss=True, category='table_body', type='value', value='#D3D3D3'), column_labels_border_top_style=OptionsInfo(scss=True, category='column_labels', type='value', value='solid'), column_labels_border_top_width=OptionsInfo(scss=True, category='column_labels', type='px', value='2px'), column_labels_border_top_color=OptionsInfo(scss=True, category='column_labels', type='value', value='#D3D3D3'), column_labels_border_bottom_style=OptionsInfo(scss=True, category='column_labels', type='value', value='solid'), column_labels_border_bottom_width=OptionsInfo(scss=True, category='column_labels', type='px', value='2px'), column_labels_border_bottom_color=OptionsInfo(scss=True, category='column_labels', type='value', value='#D3D3D3'), column_labels_border_lr_style=OptionsInfo(scss=True, category='column_labels', type='value', value='none'), column_labels_border_lr_width=OptionsInfo(scss=True, category='column_labels', type='px', value='1px'), column_labels_border_lr_color=OptionsInfo(scss=True, category='column_labels', type='value', value='#D3D3D3'), column_labels_hidden=OptionsInfo(scss=False, category='column_labels', type='boolean', value=False), row_group_background_color=OptionsInfo(scss=True, category='row_group', type='value', value=None), row_group_font_size=OptionsInfo(scss=True, category='row_group', type='px', value='100%'), row_group_font_weight=OptionsInfo(scss=True, category='row_group', type='value', value='initial'), row_group_text_transform=OptionsInfo(scss=True, category='row_group', type='value', value='inherit'), row_group_padding=OptionsInfo(scss=True, category='row_group', type='px', value='8px'), row_group_padding_horizontal=OptionsInfo(scss=True, category='row_group', type='px', value='5px'), row_group_border_top_style=OptionsInfo(scss=True, category='row_group', type='value', value='solid'), row_group_border_top_width=OptionsInfo(scss=True, category='row_group', type='px', value='2px'), row_group_border_top_color=OptionsInfo(scss=True, category='row_group', type='value', value='#D3D3D3'), row_group_border_right_style=OptionsInfo(scss=True, category='row_group', type='value', value='none'), row_group_border_right_width=OptionsInfo(scss=True, category='row_group', type='px', value='1px'), row_group_border_right_color=OptionsInfo(scss=True, category='row_group', type='value', value='#D3D3D3'), row_group_border_bottom_style=OptionsInfo(scss=True, category='row_group', type='value', value='solid'), row_group_border_bottom_width=OptionsInfo(scss=True, category='row_group', type='px', value='2px'), row_group_border_bottom_color=OptionsInfo(scss=True, category='row_group', type='value', value='#D3D3D3'), row_group_border_left_style=OptionsInfo(scss=True, category='row_group', type='value', value='none'), row_group_border_left_width=OptionsInfo(scss=True, category='row_group', type='px', value='1px'), row_group_border_left_color=OptionsInfo(scss=True, category='row_group', type='value', value='#D3D3D3'), row_group_as_column=OptionsInfo(scss=False, category='row_group', type='boolean', value=False), table_body_hlines_style=OptionsInfo(scss=True, category='table_body', type='value', value='solid'), table_body_hlines_width=OptionsInfo(scss=True, category='table_body', type='px', value='1px'), table_body_hlines_color=OptionsInfo(scss=True, category='table_body', type='value', value='#D3D3D3'), table_body_vlines_style=OptionsInfo(scss=True, category='table_body', type='value', value='none'), table_body_vlines_width=OptionsInfo(scss=True, category='table_body', type='px', value='1px'), table_body_vlines_color=OptionsInfo(scss=True, category='table_body', type='value', value='#D3D3D3'), table_body_border_top_style=OptionsInfo(scss=True, category='table_body', type='value', value='solid'), table_body_border_top_width=OptionsInfo(scss=True, category='table_body', type='px', value='2px'), table_body_border_top_color=OptionsInfo(scss=True, category='table_body', type='value', value='#D3D3D3'), table_body_border_bottom_style=OptionsInfo(scss=True, category='table_body', type='value', value='solid'), table_body_border_bottom_width=OptionsInfo(scss=True, category='table_body', type='px', value='2px'), table_body_border_bottom_color=OptionsInfo(scss=True, category='table_body', type='value', value='#D3D3D3'), data_row_padding=OptionsInfo(scss=True, category='data_row', type='px', value='8px'), data_row_padding_horizontal=OptionsInfo(scss=True, category='data_row', type='px', value='5px'), stub_background_color=OptionsInfo(scss=True, category='stub', type='value', value=None), stub_font_size=OptionsInfo(scss=True, category='stub', type='px', value='100%'), stub_font_weight=OptionsInfo(scss=True, category='stub', type='value', value='initial'), stub_text_transform=OptionsInfo(scss=True, category='stub', type='value', value='inherit'), stub_border_style=OptionsInfo(scss=True, category='stub', type='value', value='solid'), stub_border_width=OptionsInfo(scss=True, category='stub', type='px', value='2px'), stub_border_color=OptionsInfo(scss=True, category='stub', type='value', value='#D3D3D3'), stub_row_group_background_color=OptionsInfo(scss=True, category='stub', type='value', value=None), stub_row_group_font_size=OptionsInfo(scss=True, category='stub', type='px', value='100%'), stub_row_group_font_weight=OptionsInfo(scss=True, category='stub', type='value', value='initial'), stub_row_group_text_transform=OptionsInfo(scss=True, category='stub', type='value', value='inherit'), stub_row_group_border_style=OptionsInfo(scss=True, category='stub', type='value', value='solid'), stub_row_group_border_width=OptionsInfo(scss=True, category='stub', type='px', value='2px'), stub_row_group_border_color=OptionsInfo(scss=True, category='stub', type='value', value='#D3D3D3'), source_notes_padding=OptionsInfo(scss=True, category='source_notes', type='px', value='4px'), source_notes_padding_horizontal=OptionsInfo(scss=True, category='source_notes', type='px', value='5px'), source_notes_background_color=OptionsInfo(scss=True, category='source_notes', type='value', value=None), source_notes_font_size=OptionsInfo(scss=True, category='source_notes', type='px', value='90%'), source_notes_border_bottom_style=OptionsInfo(scss=True, category='source_notes', type='value', value='none'), source_notes_border_bottom_width=OptionsInfo(scss=True, category='source_notes', type='px', value='2px'), source_notes_border_bottom_color=OptionsInfo(scss=True, category='source_notes', type='value', value='#D3D3D3'), source_notes_border_lr_style=OptionsInfo(scss=True, category='source_notes', type='value', value='none'), source_notes_border_lr_width=OptionsInfo(scss=True, category='source_notes', type='px', value='2px'), source_notes_border_lr_color=OptionsInfo(scss=True, category='source_notes', type='value', value='#D3D3D3'), source_notes_multiline=OptionsInfo(scss=False, category='source_notes', type='boolean', value=True), source_notes_sep=OptionsInfo(scss=False, category='source_notes', type='value', value=' '), row_striping_background_color=OptionsInfo(scss=True, category='row', type='value', value='rgba(128,128,128,0.05)'), row_striping_include_stub=OptionsInfo(scss=False, category='row', type='boolean', value=False), row_striping_include_table_body=OptionsInfo(scss=False, category='row', type='boolean', value=False), container_width=OptionsInfo(scss=False, category='container', type='px', value='auto'), container_height=OptionsInfo(scss=False, category='container', type='px', value='auto'), container_padding_x=OptionsInfo(scss=False, category='container', type='px', value='0px'), container_padding_y=OptionsInfo(scss=False, category='container', type='px', value='10px'), container_overflow_x=OptionsInfo(scss=False, category='container', type='overflow', value='auto'), container_overflow_y=OptionsInfo(scss=False, category='container', type='overflow', value='auto'), quarto_disable_processing=OptionsInfo(scss=False, category='quarto', type='logical', value=False), quarto_use_bootstrap=OptionsInfo(scss=False, category='quarto', type='logical', value=False)), _has_built=False)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(GT(pl_data.tail(max_samples * 4))\n",
    "    .tab_header(\"Comparing our different prompts' outputs\")\n",
    "    .tab_spanner(label=\"Samples\", columns=cs.starts_with(\"arc\"))\n",
    "    .tab_stub(rowname_col=\"Type\", groupname_col=\"Sample\")\n",
    "     .fmt_markdown(columns=cs.starts_with(\"arc\"))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07a8d041",
   "metadata": {},
   "source": [
    "有点令人匪夷所思的是，对于续写任务的对数似然，不提供示例的 prompt 才是对模型输出最优的情况，跟生成式评估的机制截然相反！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c221195",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "default310",
   "language": "python",
   "name": "default310"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
